I've felt something similar before, but not in relation to poly specifically. DH has a platonic friend that I don't care for, and it would always bother me when he would go out with him. I knew it shouldn't, because I didn't need to interact with the dude, so why should I care who he hangs out...