Our increasingly inactive lifestyle is detrimental to physical and cognitive health. This review focuses on the beneficial relation of physical activity and aerobic fitness to the brain and cognitive health in a youth and elderly population to highlight the need to change this pattern. In children, increased physical activity and higher levels of aerobic fitness have been associated with superior academic achievement and cognitive processes. Differences in brain volumes and brain function of higher-fit and lower-fit peers are potential mechanisms underlying the performance differences in cognitive challenges. We hope that this research will encourage modifications in educational policies that will increase physical activity during the school day. In addition, older adults who participate in physical activity show higher performance on a variety of cognitive tasks, coupled with less risk of cognitive impairment. The cognitive enhancements are in part driven by less age-related brain tissue loss and increases in the efficiency of brain function. Given the increasing aging population and threat of dementia, research about the plasticity of the elderly active brain has important public health implications. Collectively, the data support that participation in physical activity could enhance daily functioning, learning, achievement, and brain health in children and the elderly.