I guess I object to forms of adult relationships as being called "lifestyles" at all. Never mind comparing poly people and swingers. Take monogamy. What is a "monogamous lifestyle"? There is no such thing. You can be mono and have a million different way to live-- with kids or not, homeless or in a palace, ill or healthy, gay/queer/trans/non-binary or not, living together or not, loving each other or basically hating each other, gentle or passionate, happy or abusive, having sex or not, with involved large extended families or not, in the US suburbs or in a tiny African village, in the BCE era, the 1400s or the 2000s, etc., ad infinitum, and all of these marriages will appear quite a bit different, even vastly different, and involve different behaviors and activities throughout the days, weeks and years.
I do think that swingers and BDSM communities co-opted the term lifestyle and tainted it. In fact, the word "lifestyle" as a noun is rather new, relatively speaking, and it's actually kinda stupid. What is a "gay lifestyle"? What IS a "poly lifestyle"? There are many ways to live when you're gay. The are many ways to live when you're poly. There are even many ways to live if you're a swinger or kinkster. Sex itself is only required in swinging. The sex aspect is why they needed the euphemism of lifestyle.
I associate "lifestyle" with being coined in the early "mod" 1960s, with a sex and drugs and rock and roll, maybe jetsetter, "Eurotrash," overly trendy/fashionable, self-consciously groovy way of going. Something alternative, something that didn't pertain to a spouse, 2 kids, a 9-5 job, a house and a picket fence. Something where sex was done recreationally. If swingers feel they are better/cooler than others because they just don't fuck their spouse, they can have the damn word. Polyamorous people are more down to earth, at least in my experience and opinion (unless they are just NRE junkies, which is another topic).
I think a mere one-line definition doesn't do justice to the word lifestyle. It's pretty loaded. Gays got in a lot of shit for being seen as having a "lifestyle," often associated with an "agenda," when their "lifestyle/agenda" was just, wake up, drink coffee, eat, shit, shower, get the kids to school, go to work, buy and make food, get the kids to bed, sleep, repeat, just like pretty much any damn person on the planet.
This could devolve into a discussion, not so much of the mere word "lifestyle," as a discussion of how swingers live (with group sex with no feelings allowed) and how poly people live (with multiple loves that you date and have sex with, but one-on-one, usually). That seems to be more of a problem than one mere ill-defined word.