That is fine but if she try gets in between us I guess then we will call it quits
That is actually a boundary that I hold fiercely. Any efforts to undermine/control/stop/snoop/meddle with any other association I have will be met with an immediate "let's talk". I apply that rule to all of my associations, including a partner I might be living with or married to. An attempt to interfere with another one of my associations shows that there is some high level insecurity going on, and likely some clinging to monogamous traditionalism, which doesn't line up with how I'm trying to live my life.
I hope that you and your husband do decide to stand by the boundary you noted, and apply it to yourselves as well. If we are all living our lives as free adults, no one relationship should have the power to lord over another.