Don't worry at all, my son plays with his sisters toys and she plays with his toys.

He may get teased...he may also not care! Only worry about that if it actually happens.