Living with muslims.

Well, i live in Northern Europe, and i live in a studentapartment. Everyone who lives here is foreign people, and many of them are muslims from arabian countries and middleeastern countries. Im also gay, so the thing is, is it totally unfair to them that i don't feel safe here? (I obviously don't talk about skincolor, i absolutely hate racist people who judge people based on colors, plus all of them are from the middleeast who live here) But I have talked to people and they said that its not racist at all that i think so, because they DO have deathpenalty for being gay where they come from. They understand me very well. What do you think? Is it weird that i feel that way living with muslims all around me?