My major problem is that people see poly and immediately confuse it with slut.
It's important to keep context in mind.
People who are in non-monogamous relationships are prone to be more sex-positive and sexually expressive than their monogamous counterparts. I find it reasonable to presume that most people get into polyamory because they want to have more fulfilling sex lives
and romantic lives. People going on this type of journey are more prone to discovering concepts like sex-positivity, kink, ethical slutdom, etc., so it's not unusual to find this kind of personality being associated with poly. Thus it is not unreasonable to lump the group of us together as "sluts."
Add to that the swinging crowd. From an outside perspective, it's not unreasonable for people to confuse them with poly folk. I mean, they're in a relationship, having sex with other folks, everyone seems ok with it. They look poly to someone who doesn't really know what they're looking for.
I found myself technically qualifying as polyamorous, but it never had anything to do with how much sex I could get; it was a philosophical change unrelated to sex drive. But let's be realistic, folks. Probably most people out there practice non-monogamy at least partially because they wanna get LAID! Judging from the madness on this forum, I'm going to go out on a limb and say that coming to poly from a non-sex-drive motivation is not the norm.
I have had people tell me, "Prove it, let me be your boyfriend." Obviously that was the end of that.
That's just a retarded person who demonstrated this to you very clearly. Count yourself lucky. Most people aren't quite that damaged and are harder to spot.