Going to a co-ed birthday party with a bunch of friends

My one of my co-workers is having a birthday party at their home and invited me. A bunch of other people from work will be there. I don’t like being in social settings around men and I don’t like to drink so I’m not sure if I’m going to go. If I don’t go, I don’t wanna leave my coworker out, but if I do I can’t imagine having fun and I’d be uneasy the whole time. Advice?