Well if he is happy and that is most important thing then I wouldn't worry too much. Some people just don't need as much human contact as others. I would ask the teacher again if he/she feels your son is happy as it is bothering you, how is going with his social skills, is there anything you can do to help, should you worry etc.
As he is the youngest in his class his maturation development might just be a bit slow. Given time he might catch up and be more social or else he might always be a loner. I don't think there is necessarily anything wrong with that provided your son is happy. But perhaps you could get him involved in some extra cirricular activities like sport, tae kwon do, or he might enjoy something like chess I just wouldn't push him too hard to make friends as we are all different.
My eldest child is now 11 and she was very young in her year. When she was in Year 1 (aged 5) we were called in to school as the teacher was extremely concerned that she was very quiet, a loner etc etc ( and an underschiever ... but that was everyone's lest worry at the time). Every year we were told she was a bit of a loner "happy in her own company" and she drifted from the periphery of several groups of friends, often alone in the playground. As she went through her junior years she was much happier and filled her lunch-times in clubs, or helping the teacher. Suddenly in Year 6 ( when she was 10) she settled into a steady group of friends and we were told she was "chatty" for the first time. She is still quiet at school ( never at home) but has good friends. There is hope.I had to realise that she was quiet happy as a loner .. the world needs all sorts .. I am gregarious and it took me a long time to realise it's a perfectly ok to be like that. I also had to learn not to interfere so much. I used to ask every day "Who did you play with today ?"... to a certain extent I created the problem. It wasn't much of a problem to her. As it happened, she hit puberty first ... maybe that helped her to be more confident in making friends as she wasn't the one catching up then. Please don't worry too much... for her it sorted itself out with maturity.