Children frequently hate math because they have a shaky foundation in the basic skills and find it very difficult to do the required work. Fill in the holes in your son's knowledge of math, and he should begin to enjoy it.
It's great that you and the tutor are working closely with your child. To maximize the benefit of this help, you must coordinate your efforts. Do go beyond helping him just complete his homework to helping him improve his basic skills and learn how to solve story problems.
In helping your son, you and the tutor should avoid relying solely on drill type work, as it can kill interest in math. Instead, find games that will let him use the skills he needs. For example, Dominoes and Twenty-one are fun to play as well as great ways to teach addition and subtraction. Learning stores carry many enjoyable educational games that will make math more fun for your son.
Retaining your son in fourth grade does not offer any guarantee that he will master the skills he needs in math. Instead, he needs specific help on his weak areas now.