I guess I'd tend to see the black and white (that's what us chaps do you know). If you're concerned about the safety of your boy, then the choice is simple.
If your husband values the safety of his lad he'll agree.
Bottom line, if you have rules when it comes to looking after your kids, then all minders should comply, or not take care of them.
You two are the parents, you have the most to lose.
You've also got to be willing to give up your nights out when grandma would usually have him if she digs in her heals.