I know, right? Non-poly people often do not understand this aspect of dating around. It's HARD, and often an open, communicative marriage is safe and easy to retreat into.
You don't sound like you have a problem, really. Enjoy your wonderful marriage, and good luck with the other partners...