I'd sum it up like this, I think -- some people who go the marriage-and-kids route choose to make their nuclear family their whole life (not counting work or family-of-origin). That's ok! Some other people choose to make time to have a life outside of their nuclear family. That can include other friends, hobbies, classes, exercise, games, social events, or any number of things.
I think we can all agree that any working mom or dad who also works it out with their nuclear family such that they can go to a class for one night every other week, or go out with friends for coffee on occasional weekend afternoons, or whatever, is not therefore a bad partner/parent. Well, just substitute out "date" for any of the activities I've been describing above and bam, non-monogamy (and possibly poly if feelings are involved).
Now, obviously time will be limited, and any new partner who wants to get more than passingly close to the person in question will need to be able to be ok being around kids so that the relationship can extend beyond the small amount of time allocated for dates. Hopefully, with time, the new person will become close enough to the family to have both the trust from the parents and the personal interest to help out in some ways, such as babysitting from time to time, which can help the parents keep their own bond strong.
So -- are you and your wife the "nuclear family only" types or the "maintain an outside life" types? If the former, maybe let the idea of poly go. If the latter, maybe see what you could let go of in order to put dating in its place instead.