Yes, part of the problem is that the whole house, including his room if she's spending most nights there, is going to be THEIR space instead of HIS space.
I want to ask him to limit the number of nights they spend together, or to spend as many with me as he does with her, but that feels selfish...