I would give the baby groups a miss for now, neither of you are enjoying them and he's no need for them, any company is ok at his age, it'll be a while before he needs to mix with his own age group to develop social skills. Baby groups are more for the benefit of the parent but your not enjoying them either so no benefit to either of you.

This is normal for babies, its just a phase he will grow out of it."Once 1 baby starts they all start" You need to teach him that its ok for other babies to cry, Reassure him and cuddle him and talk to him calmly.