Your pheremones fit into each other's brains differently depending on your sex. Of COURSE those things are going to fundamentally affect attraction! Color, on the other hand, is cosmetic.
It's not comparable.
I recently met a Chinese woman who told me she should would never have a relationship with a white man because of the way white people smell.
I wonder, is she racist, or is she simply noticing that there are differences that affect attraction?