When I was a vegetarian, I used to enjoy the smell of meat being cooked on a grill, and it didn't make me any less vegetarian.I REALLY don't want this feelings, because they are more a burden than something to enjoy.
Maybe it would help if you thought of your feelings like that.
Another thing to think about is that your feelings are not what it would really be like to be in a relationship with this person on a day-to-day basis.
If that isn't helpful, then just disregard.