Ouch, I feel for you! It sounds like you two have a lot on your plates, and maybe he is even a little depressed?
If he seems to be in a place where "yes" is the only answer he wants from you, try this one: "Yes, I agree you need some romance and dating and sex in your life, but with our toddler...