Your situation catches my attention because,
is almost exactly what I said to my wife (switch "male" with "female") and that's what got the ball rolling for us.
Admittedly, I feel more comfortable, or "less cheated on," when she is with women, and (sorry if this sounds ignorant) maybe the same...