Hey SaBo, no, it is not selfish. Everybody has feelings.
The other night my mono boyfriend and I had this same talk, in which I told him of my feelings and how somehow even though I'm poly and don't think I can change, it still freaks me out the idea of him dating anyone else at the same time...