In terms of number and letter recognition, your child may not have developed the perceptual (that is, visual memory) skills that allow him to hold on to these visual images. Or, he may get the concepts underlying the images, but have little interest in the symbols themselves, as they have little relevance. It's likely that this apparent visual memory "problem" is not an indication of anything serious and should get better with time and natural exposure to numbers and letters. It is possible, however, that even at this young age your son is exhibiting symptoms of a weak visual memory that might signal the presence of an underlying learning disability.
I think a cautious but optimistic approach is advisable. Give your son many opportunities to be exposed to numbers and letters that involve many of his senses, and try not to "test" him. Make sure to give your son lots of chances to interact with blocks and other shapes that aren't associated with numbers or letters. Playing memory "games" by taking away one of the blocks and then having him pick one (from two) that you are holding will also help him learn to keep things in his "mind's eye."
Finally, even though your pediatrician feels that your son's vision is okay, the tripping and falling could signal some visual problem. Again, this is not likely, but it's important to pursue if it doesn't improve. Some G-and-T kids seem like "klutzes" because their verbal skills outshine their visual-motor skills, and parents worry about this difference instead of just waiting for the gap to narrow as kids get older -- and it usually does. If things don't seem to get better, then seek out the advice of someone who specializes in the assessment of early learning disabilities. You can ask your pediatrician for a recommendation.