I have been with my husband 26 years and of course he always says, "Sure you can have a girlfriend, if I can watch," but I'm not certain that is what I want.
If you're an exhibitionist and something of an actor, I could see performing sex for your husband to watch. Like, if it was his birthday, hire a sex worker and do sex with her as he watches. But I believe that would be an empty experience for you.
You want a real relationship with a person, which includes sex. It looks like you've brought it up a few times (?) and he makes it into a joke about voyeurism. Maybe you could tell him you'd rather discuss this seriously without the jokes. Lots of coupled people who are curious about poly broach the subject with "subtle" jokes, but there comes a point when the jokes have to stop and an actual discussion has to happen.
I'd like to explore this option with just another woman and myself to see if a poly relationship is something I'd even like to do. I have no clue how to approach this from a physical, emotional and health aspect.
Well, you have to be emotionally drawn to someone, for it to be polyamory. That's the amory part. Then, usually, for adults, sexual attraction follows love or fond feelings.
As far as the physical part, it goes like it does with guys-- flirting, kissing, holding each other, arousal, sex... you know, the usual.
By "the health aspect," do you mean physical health, as in avoiding STDs, or emotional healthy, as in avoiding emotional difficulties for everyone, you, husband, and your prospective dating partner?
I don't want to introduce another woman if I am not interested because I don't want to upset anyone if it doesn't feel right.
I am not sure what this part means. "Introduce a woman," you mean, start a relationship with one? Introduce yourself to her? You're not introducing her to your current relationship, or adding her, or including her. You will be dating her. Your husband won't.
It's impossible to avoid some upset when dating. Generally there will be some rejection, some breakups, etc. Even if you get along great with someone there will be upsets from time to time. "If it doesn't feel right" to be with someone, you might want to get counseling and repair things, but if you can't repair disagreements, you'll need to break up.
Lots of first dates go nowhere. Sometimes there will be two or three dates and then you decide, "Nah, it's not working. Thanks anyway." It's doesn't have to be upsetting when you realize you just aren't interested in someone. Dating is just sampling, to see who clicks with you.