Yes and no. Being worried is indeed perfectly normal. However, before I threaten to my partner to take the kids away, I need much much more than some worries. I nearly need proof. I need to see that it is indeed affecting them in a bad way - being in a way worse then losing the mother.
His not having discussing the affect to the children makes quite clear, that it is not his worries about the children, but his worries about himself what made him act like that!
He even can declare to not being willing to continue the relationship. No one here would blame him (at least not badly). But taking the mother away from the children? Sorry. No go.
Additionally, children are more flexible than you expect them to be, at least imho. If they still feel loved and valued by both parents they accept a lot of changes in the environment.