Having sex, as a couple, with another single person, say, in a swinging sense, is one hurdle. Even that can cause uncomfortable feelings, if not during the actual sex, then a day or two later.
But we have swingers come here all the time who have managed to have other sex partners for years, and remain basically anonymous, or just friends, sharing bodies only, maybe a few shallow friendships, just to finally seem to fall in love with someone. Then they are thrown for a major loop, lose their shit, everything goes awry and they need help.
That said, polyamorists hardly ever have group sex (3 or more). (You can be poly and a swinger, but they are 2 distinct practices.) Successful triads, where everyone loves and desires each one equally, are extremely rare. Most poly people date as individuals. If you've been coupled for a long time, with your partner, or mono in general your whole life, you will need to develop an amount of independence from each other, and learn to respect the privacy of each, plus the privacy of the prospective dating partners. That is the key. It's not easy for most people.
Reading a really good book about polyamory can help. I highly recommend Opening Up, which also covers swinging.