My neighbour fancies me but.....?

He is nice, but we are both married. Neither of us have done anything other than chat, but the brother of his wife he lives opposite us and is giving me daggers, but not his brother in law. All we do is chat at the door, so why am I getting the stick. BTW I would never cheat on my husband, but it's clear he would cheat on his wife. Should i confront the brother?