You should certainly provide him with the opportunity to be exposed to colors, numbers, shapes, etc. Allow him to play with paints, crayons, clay, and natural objects. Go to museums and zoos and libraries. Read to him every day. But don't drill him -- that's certain to take the joy and fun out of learning.
He will also be less likely to answer your questions if he feels he will be criticized for being wrong. It is much better for him to come to you one day and say "Mommy, this crayon is green," having discovered it on his own, than for him to spend time having someone ask him repeatedly which color is which.
The same goes for the numbers. Certainly provide games, songs, and activities that include counting, but do not test him on it. There will be plenty of time for structured learning later on. Let him enjoy being a kid for now.