Outlooks on sex are a very personal thing. In time you will find your own. It's good that you're exploring as many aspects as possible - you'll understand better what feels 'right' to you and not have to theorize.
My opinion - 'casual' sex is no different than casual dining
At least in terms of where it fits in a relationship. The only discomfort I would have is making clear everyone is on the same page with health concerns. That's a given.
But what matters most is that you and your SO are talking and sharing your feelings & experiences about how it evolves. Because it IS likely to evolve over time. Both your attitudes about sex will change as your life changes - with stress, possible children, social/professional circles - lots of reasons.
I'm one who happens to believe that sex is 'just sex' and as long as it's kept in that perspective it's a lot easier to talk about where it fits in. It disconnects a lot of emotional reactions that cloud good judgment.
Where a lot of people get into trouble is when they can't disconnect sex & love - then the waters get very murky.
It's a perfectly natural biological desire and to me should be treated the same as where we'll eat tonight & what's on the menu !
And not to imply for one second that sex can't be MORE between certain people. But that's another subject altogether.