So it's already "not equal."
Triangle shapes don't have to be "equal" to be "stable." Can easily see that in geometry -- It doesn't have to be an equilateral triangle. Right triangles and isosceles triangles have stable bases/foundations.
Sounds like she's become more confident and secure knowing that even if you and BF shared sex without her, she's not being replaced or rejected.
So, why not just drop the group sex? You don't esp enjoy it, and GF is already asexual and only sharing sex periodically. It doesn't have to be group sex.
I could be wrong but it sounds like he just doesn't want to give up the novelty of group sex and saying whatever just to hang on to it.
I am concerned about you living in a home with a toxic person. Can you live elsewhere? I get driving roomies around sometimes and doing household stuff beyond fair roomie chores if that's part of how you pay your room and board. Taking on extra jobs.
But do you HAVE to be THERE? If you are so scared for him to live there alone? The solution is to encourage him get out, not go stick you in the mess with him.
GF seems to know her personal boundaries are best served keeping away from the toxic mom. I suggest you think about doing same.
Galagirl