Sure, if you're happy for him to have that access. If you're not, tell him "no" and accept the outcome.
Your boundaries are what you make them, and the consequences are on you, not on anyone else. If you want privacy in your communications with others then that's where you draw your line...