Would I (F) be in the wrong for kicking out my (25M) boyfriend over the clothes he's been wearing?

My boyfriend recently started working at a new place about a month ago and he seems to be getting along pretty well. Recently I've noticed that he's been wearing pretty feminine clothing like blouses, skirts, and dresses. He's been doing this primarily in the house but I'm getting worried that he'll start wearing it outside, especially at work. I confronted him about it and he got very defensive but he did end up saying that he's never done this before and doesn't know why he's doing it. I'm really frustrated and I want him to stop. If I tell him to either stop wearing it or get out, AITA?