If he's got a wife, he's married, and that right there is a hierarchy. It might feel better to you if he owned and acknowledged that.
Some long-term couples do "we" talk, like they are some kind of CoupleBlob.
When you say:
... you are asking him on a date, not her.
And him talking like...