I don't think many 4 yo feel remorse. I have a friend who is a child psychiatrist and she told me empathy and genuine remorse are quite high order skills that don't come till teenage years. I think you can teach children to say sorry before this happens and teach them that certain things are wrong, but remorse isn't often genuine because they can't really put themselves in someone else's shoes yet. So I imagine keeping on telling him "No, it's not OK" is good. The punishment that works with children is often different for every child. One of my children (older than 4) is money orientated and cutting pocket money works, one reacts well to having no computer time and one hates being sent to bed early. If he is continuing to steal maybe you haven't found what matters to him yet. Have you tried making him help clear up the mess ? Good luck. I'm sure you will manage to nip it in the bud.