IMO, you're not in a poly relationship. Your husband is simply having his affair in your home rather than playing away.
As a couple, the two of you should have negotiated who wanted what, and how each of you, and/or another partner, would provide that. Polyamorous relationships are negotiated...