What were really little issues that could have been avoided by having a set of rules in place up front, wound up seriously damaging a family relationship that took many years to be made whole again.
I know this is a radical idea and all, but I would have just talked about the issue when it came up instead of letting it build to the point of resentment and anger. Been there, done that, really don't want the t-shirt.
As Ceoli said, house rules are not the same thing as relationship rules. However, pretty much everyone that I am friends with doesn't have "house rules" because they don't tend to invite people into their home that are likely to do things to damage it or disrespect the people in it. If I let you into my home, it means I trust you. Why would I let you in if I don't trust you? If I don't know you well enough to trust you not to track mud onto my carpets or terrorize my cats, I probably shouldn't be letting you into my home in the first place. If it turns out I misjudged you, well, I'd like to think that I'd catch that before you did something like driving a car through my living room and I'll ask you to leave. If it turns out you're a basically decent person but we have mismatched expectations, an open line of communication should be all that's required to solve that. If you don't want to communicate with me about a problem I'm having with how you treat my home, well, we go back to the first solution and I kick you out.
If a person honestly cares about his partner, is considerate and does things with his partner's best interests at heart, rules are unnecessary because he will want to be caring and considerate automatically. If he is not caring and considerate and does not have his partners' best interests at heart, a "rule" won't stop him from doing what he wants to do anyway.
As to the problem of needing time to see if the prospective partner is going to be problematic or not, I like to solve that issue by building a friendship with someone before dating them. I don't always do that, and quite often when I don't, those are the ones that blow up rather spectacularly.
One of the benefits of polyamory is that everyone comes with references. I can build a pretty good idea about how a prospective partner handles certain issues, how he will handle a breakup, and how compatible we might be by getting to know him AND his current and past partners. I can watch him and see how he interacts. I can check out his track record. This is a very good indication of what I can expect from him in the future and tends to answer all those pesky questions that these types of primary-protecting-rules are supposedly designed to protect the primary relationship from finding out the hard way.
With my current partner, we've been together for several years now and he has his own history of being caring and considerate of his existing partners' feelings. He has shown me time and again, before we ever started dating, that he has a very high standard of treatment of his partners. It is completely unnecessary for me to place restrictions on his behaviour with potential partners because I know that he will not get into a relationship with anyone who could "threaten" what he and I have together. He is interested only in women who have an equally high standard for how they treat their partners and metamours. I trust *him* enough to make decisions that will not harm me or his relationship with me, so I do not need to give him any rules as though he were a child who is not capable of deciding for himself who is toxic and who is respectful of his existing relationships.
If my relationship with someone is so fragile that another relationship has to have limits on it, dictating what their emotional entanglement should be before anyone even knows what it wants to be, before even having another prospective relationship on the horizon, then we shouldn't have an open relationship.
And if my new metamour actually has the potential to damage whatever someone and I have together, than she's bad news and we have way bigger problems than a few rules like "don't stay out past midnight" can handle.
As Ceoli has already said several times, this is not the same thing as setting personal boundaries, this is participants in an existing relationship setting up the structure of a yet-to-come relationship without the input of the yet-to-come partner.