Sit down with BOTH of them together and explain what happened. Be firm and in control and do not deviate from what you KNOW happened, if either of them try to paint you as a liar or make excuses.
What happened WAS sexual assault - by your gf's husband on at least two occasions, perhaps three (not totally sure about the second incident, as you don't make it clear, OP).
What happened was NOT OKAY and you have every right to feel angry, scared and upset. Naturally, you do not want to "upset the applecart" and possibly lose your relationship (if your gf doesn't believe you or sides with her husband), a good friendship with the husband who's been like a brother to you until these recent episodes, even your home! But remember, this was NOT YOUR FAULT and it's imperative you address his behaviour so he doesn't think it's okay to try this again - either with you OR anybody else in the future.
Since you say you all have been in this relationship/living situation for years and nothing like it has EVER happened before (you do appear shocked/shaken), it is JUST possible that the husband misinterpreted your decision to stay in their bed after he'd returned, and thought he'd try it on (make an "advance" or three) to see where it'd get him. He might've had fantasies of a threesome or the like, and even though his approach was totally inappropriate and wrongwrongwrong on so many levels, it MAY explain it. Especially if the wife/your girlfriend was the party touching you in that second instance. ASK them what is going on. It is still inappropriate/assault, when it all boils down to it, but there is a slim chance he/they might have thought that's what you were looking for, or that you'd be up for it.
Hopefully, your reaction (or rather, lack of reaction) i.e. leaving the bed without a word, should be enough to alert the husband that you are NOT okay with it, but I still think you need to address this with BOTH of them. If I was your gf, I'd want to know that my husband was trying to grope my other partner without her consent. There is no way she can make an informed decision without full knowledge of what is going on in her marriage bed under her nose. Once you hear both parties out, you'll be in a better position to decide what you yourself intend to do. It's not an easy situation and I'm so sorry this happened to you.
