AITA for telling my dad that if he ever called my son a “girl” again I would beat the shit out of him?

I have a son who is 3 years old. He loves Thomas the tank engine, my wife knitted him a train blue hat and my dad called him the little girl. My wife said it was rude but he told her to watch it. I said he was disrespectful towards my family and he said he was just saying what he saw. I told him he was being intentionally rude and to behave and he repeated what he said to my wife. I said he disrespected me and my family for the last time and I grabbed him and I hit him across the face and I told him that if he called my son a girl again I’d beat the shit out of him. He went to tell my brother and my brother believed me and he said dad was in the wrong and my sister-in-law (who is achsual) said I was a monster for hitting my father. He won’t listen and I won’t apologize, am I the a hole?