Halloween can be a terrifying time for little ones. Your responsibility isn't to make Halloween the great holiday you enjoyed as a child. But you can help him moderate his fears over time and find some enjoyment in this celebration.
First, get a baseline on what it is that scares him the most, some, or a little about Halloween -- the scary masks, remembering a scary movie or video about Halloween? Once you know the details of his fears and "say them back" to him, he'll know that you are on his side and will not make him feel "like a baby" for being scared about these things. Obviously, you should never blame or shame him for being afraid of some things associated with Halloween.
Brainstorm some ideas for activities he might like to try as part of celebrating Halloween. Maybe he doesn't want to go out to trick-or-treat, but he might like to give out treats at your house to kids who come dressed up in costumes that don't frighten him. Consider inviting some of his buddies over for a Halloween party, making sure that none of them wears a scary costume. They could all use some greasepaint makeup at the party and make themselves up, as opposed to putting on masks. Your son might even be able to make some headway in terms of his fears if you allowed him to pick out a costume and a mask he would like. Standing before a mirror in his costume (maybe even a scary one) might make him feel less like his "scared self" and more like a "Halloween boy." The goal here is to do something to celebrate the day that makes him comfortable and happy, not vulnerable and scared.
If none of these ideas seem workable, it may be a good idea to celebrate the holiday in a different way. Go out to dinner and let him have his favorite dessert, or rent a video and watch it with him while someone else in the family attends to your trick-or-treaters. Don't make his conquering these Halloween fears and your wanting him to enjoy it like you did the priority this Halloween. Give him a chance to participate or not, based upon his comfort level. There'll be many more Halloweens for him to enjoy as he rids himself of these very real fears.