I wouldn't say I'm afraid of being secondary, but moreso I do not want to be in a situation that I feel is being run by someone else.
I have dated another person who is married, and I have no issues with his other people because our definitions are clearly set, and they all live far enough away...