That would be VERY difficult. We are blessed by 2/3 of our triad working remote (so no issues there), and Taylor having a job that really doesn't care. (He doesn't advertise at work AT ALL, but they are pretty open and not invasive in privacy in general, so he also doesn't feel he has to like HIDE, which is nice.)
So far, work has been a blessing for us. 2 of us come from fairly poor backgrounds. We all make solid salaries, but not like BIG bucks. Going from 3 households, to 1 household on 3 incomes, is really opening things up for us financially, both individually (everyone gets 30% of their salary to save or spend) and collectively (the house, trips, eating out). Before this I couldn't have *imagined* being able to save 30% of my salary, and that's while doing LESS fun stuff than we think we'll be able to do.
The problem, as the OP says, is how it all goes back to public perception. There are a LOT of jobs that would try to push you out if they discovered the lifestyle, and that is SCARY.