There are a number of possible reasons for this. The first thing to consider is whether or not he is hearing adequately. The fact that he seems to be able to understand what you're saying to him makes a complete hearing loss less likely, but it is certainly important to make sure that he has his hearing tested to look for a mild or moderate hearing loss. It also makes sense to look at what other efforts your son makes to communicate. If he is able to use gestures and pointing and crying to communicate his wishes, then that is further evidence that his understanding is good.
I would recommend that you speak to your pediatrician about this and have his hearing evaluated right away. He will probably have to start some form of speech therapy. At this age, this may consist of increasing his exposure to books and language on a day-to-day basis, but it could also involve more intensive speech therapy. In general, speech and language delays in children are very amenable to therapy.