It is not wrong because it had nothing to do with you, and everything about the other person's perception and self-esteem. If it wasn't you, it would probably be somebody else who they got to push away.