My son, who is pre-med, goes into paroxysms whenever he hears this theory that children should be basically placed on a low-fat diet by the time they’re two years old. Except in very rare cases, this isn’t true. As my son points out, young children need fats to help with neurological development. Whole milk is certainly not going to hurt them, unless they’re lactose intolerant.
Personally, I prefer whole milk yogurt to the low fat stuff. It’s more satisfying.