Is it automatically homophobic to say that somebody "Looks gay"?

I don't want to color the answers by giving too much context. Obviously if you say it in a derogatory manner, it's homophobic but that's not what I'm talking about.