Ok. This is a question I see often. You need to decide if this is a hobby or a business. In the beginning, one of you (at a minimum) needs to be there, be physically invested for at least 100 hours a week (if you are open that many). If the doors are open, you need to be there.
Best scenario: You both invest your time and be there. This is critical. When you start a business, it's not "Let's give it a go." It's more like "All our chips are in". You need to put yourself into a corner where the only option is success. By both of you not investing your full time into it you are already telling yourselves you have doubts. And honestly, in the beginning, you may not be able to afford competent help. Remember, someone willing to work is not competent help.
Every decision you make concerning your business needs to come from a certain psychological mindset. Anything outside of that mindset is just a way to buffer yourself, to create an excuse should failure arise.
Do what you must to succeed.
Acceptable option: I would have you in charge and there and your husband can make the transition slowly. How old are your kids? Put 'em to work! Realistically, there's a lot you can do as a family business that you can't do otherwise. Kids can wash dishes, clear tables, and I know kids who can pull shots at 5. It's never to early to put 'em to work!
If they are very young, it's something you'll have to deal with. It's best they have their mom around, but if you start the business, one or both of you needs to be there until you have established a steady clientele and developed competent and well-trained employees.
The reality is, when you take on a new business, there need to be sacrifices. A business is like an unruly child. Be smart, be firm, and it will listen to you. If you've managed four children, this will likely seem easy. No sleep. Hints of mania. Frustration. Reward.
In the end, you'll get out whatever you invest.
Develop a strategic game plan and stick to it.
Know that owning a coffee shop/ business is far, far different than managing one. Make sure you understand everything it entails. Being great at the former does not mean any success at the latter.
Both of you need to be all in on this emotionally and psychologically or it will not work.
Be smart. You can do it.