Then given your carelessness and inconsistency in the past, you will need to be especially firm in the "nothing going on" direction. No breaks/lunches together, leaving together, arriving together, etc. If you give people even a little bit, the rumours will persist.
Or, accept all this as the...