Rory, this is not why I said this, and I am sorry if I offended you.
Remember, A issued the invitation first. If he wasn't interested, he wouldn't have called. B can call too. Then A can play hot and cold, if he wants to.
And no, you never owe some one sex, ever. But if this is a sexual...