I quote these to highlight how Whiskers doesn't show emotion. Which is quite typical of males! Maybe with your other partners it's different. Which is why things are going well with them.
But Whiskers is maybe just enjoying "fun" with you. Sex, good times, light conversation. You however, want to be fully human with him, deeper than surface "fun." You want the whole package.
You say he "makes" your nervous. But no one can make anyone feel anything. Take responsibility for what is yours. You feel uncomfortable around Whiskers because he doesn't show a full range of human emotion around you. However, he may never feel relaxed enough to get to this point. Especially since now you only see each other once a month. He may have chosen you because you have 2 other partners, so he has figured his role is just to be the "fun, casual" partner. That is what he wants. Can you live with that?
Personally, if I had deep relationships with 2 men, I'd be OK with having a third guy, a more casual sex partner, who was fun or funny or really cute eye candy, or relaxing to be around, and the sex was AWESOME. I really enjoy sex while it's happening. And it's a great stress relief that can make my whole week feel brighter. I might feel a tinge of regret this great handsome well-hung, intelligent man only wanted me for sex and fun, and wasn't interested in going higher on the relationship escalator, but I might adjust if it seemed worth it over all. That's the good thing about polyamory, not every partner needs to be the "whole package." We can meet them where they are at, if other things in the relationship make it worth it.
You weren't "stupid." You aren't a "fuck up." Those are self-bullying words. Why be your own bully?