Passing "Straight" Privilege

As the title suggests I am curious what are people's opinion of gay people who are able to straight pass. Even though I came out about a year ago as bisexual(now identify as gay) I only came out to my close friends and family. With most other people no one assumes my sexuality. I am a pretty private person in real life, and I have a deep voice. I am pretty average in masculinity. Since I do not conform to most gay stereotypes when I was closeted I was able to remain private, and not have anyone suspect my sexuality. However this also made being in the closet more isolating as I feared people's perception of me would change based on stereotypes of gay people. Thankfully when I cameout most of the people in my life were accepting of my sexuality. But even now some people in my life including my Mother still think it is a "phase", and ask me if I will eventually get a girlfriend. I also have awkward encounters where I have turned down girls who have flirted on me, who assumed I was straight. Even other gay men(who are more stereo typically gay) have told me I don't seem gay because my mannerisms are more masculine. I understand that not everyone fits the stereotypical image of a gay people, but it is frustrating to see how people try to put me into a box. Two I don't feel a person's gender identity necessarily corresponds with their sexual orientation. It gets quiet irritating to have to explain people my sexuality because I feel it should be my private business. I admit I have been privileged in not having had experienced many people question my sexuality, a lot of harassment or discrimination as a gay person because I conform to ideas of a cisgender male. I also realize that more stereotypical feminine gay men face prejudices from both straight people and within the gay community. But I do think it does get frustrating at times that I have to explain my sexual orientation to both straight and gay people who will assume I am straight. I don't feel in any way I should be proud of being able to straight pass but in some ways it made my life less complicated. With the exception of being asked about relationships, I remain unassuming to most people. But that has also made it harder for me to identify with the gay community because I don't feel I conform to stereotypes.