I think it's fine to continue as things are but you don't sound like you want relationships where he does, how will that work long term? I don't know if I would move in with someone, get married, etc to someone who had a different poly style than me. He's already pushing the boundaries, what if...