Q: I may have accidently offended someone help

So Im straight, married 3 kids and i work for a big corporation in the bay area. One of my smoking buddies is gay. He talks with a heavy lisp. We work in different departments but I guess he got pulled in on a project that I'm working on. We talked about it downstairs smoking. So on a conference call later on I heard his voice on the phone. So the next time I saw him while I said hey I heard you on the call. Sorry for getting in dragged in to this. He responded: it's because of my faggity voice isn't it.I said well I heard your name too. I really didn't know what to say. He went on saying: I knew it [his name] you have such a faggity voice.I tried to divert the subject by saying I'm an immigrant and I tried really hard to not talk with an accent but it comes out when I'm drunk. We laughed. Then when he left he said thanks for confirming what I already knew. Again I didn't know what to say besides sorry.Now I really don't care if he fuck dudes or purple aliens. I'm 35 and he's maybe 45 50. So maybe he's been through some tough times. I really want to say to him hey man, you should be proud of your "faggity" voice. (I'm not gonna say the faggity part) who the fuck cares if they can recognize your voice, it's your fucking voice. Also I really wanna give him a hug and say that I never judged him and people who do can go fuck themselves.I don't know, I'm just a little shocked and scared that I may have upset him. I know it's not my fault but I feel.like I should reach out to him somehow. I don't want to have to move to a different smoking section. This one has the perfect amount of shade/sun ratio.Looking for help as to what I should do or say.Sorry if wrong subreddit.Edit. I'm married with 3 kids, not married to 3 kids.