If you enjoy his company and it's all going well, and he treats you well, go with it! That said, there are definitely a few things to think about:
1. It sounds like you have some self-esteem, trust, and insecurity issues that you need to work on, whether or not you pursue this relationship...