Well, from my limited perspective, not having been there in your shoes, I can say that you may need to explore the boundaries with your wife. I would also think that if you were uncomfortable with what was going on you should have told him to stop! The other guy can't read minds, I'm sure of...