But another part of me feels like for whatever reason I am just too invested in him to be able to have good boundaries if we were to go back to just being friends.
At minimum -- you don't have styles that mesh. So why keep banging head on wall over it?
If you aren't treated well here how you want to be treated? Whether he does it on purpose or not, you still get dinged. It's ok to step back so you stop getting dinged.