The nauseating "gay but homophobic" label

I want legitimate thoughts on this because it makes NO sense to me. I am a 23 year old gay male, 100% out to everyone in my life, 100% comfortable, not ashamed to be a part of my community whatsoever. But every time I make some gay friends the same shit happens: they invite me to a gay club, I opt out and simply let them know I'm not a fan of gay-specific venues, and then all of a sudden I'm "just not comfortable with my sexuality", "one of those homophobic gays", etc. WHY. I don't get it. I don't like gay clubs, I don't get along with anyone at them very well and I've never had a good experience at one. Why does that make me "not comfortable with my sexuality"? Not the case at all. I'm as comfortable as they get. What I'm NOT comfortable with, however, is putting myself into an environment in which everyone around me thinks they can freely touch me and take me home with them like I'm a fuckin build-a-bear. From my perspective, that makes me uncomfortable with that specific type of social situation. But from the perspective of my fellow gay friends, it makes me uncomfortable with my sexuality and even HOMOPHOBIC. Okay. A young woman at a straight club endures the same issues but nobody's angry at her for feeling that way. So I wanna know, where's the logic behind this.. Someone give me insight. Because this makes NO sense to me. At the end of the day I'm not stressed about it because people who are that quick to tell me I'm hateful and disrespectful to my own community for not wanting to be in these types of situations, weren't ever my friends in the first place so that's whatever. But I'm still curious; those of you who genuinely believe that any gay man who doesn't enjoy going out on Thursday nights in short shorts and getting borderline molested for free drinks is homophobic or uncomfortable with being gay, please enlighten me.