Background: The importance of youth physical activity (PA) for adulthood PA, performance, and health was retrospectively evaluated. Methods: A total of 258,146 participants (49% women), aged 19–70, with a first-time health-profile assessment between 1982 and 2015, provided self-reported data on current perceived health, PA, lifestyle, and physical education class participation, and PA outside school hours before age 20. Data on anthropometrics, blood pressure, and estimated maximal oxygen consumption (VO2max) were obtained. Results: Women participating in physical education class, compared with those who did not, had significantly lower OR (range: 0.81–0.87) for perceiving poor overall health, general obesity, and high diastolic blood pressure after adjustment for potential confounders, and increased OR (range: 1.17–1.23) for exercising regularly and a normal/high VO2max in adulthood. For men, the ORs were significantly lower (range: 0.66–0.86) for poor perceived overall health, general, and abdominal obesity. These associations were seen for participants up to 70 years. Increased PA outside school hours revealed even stronger beneficial associations. In joint analyses, both youth and current PA were important for lower OR of poor health and being obese in adulthood. Conclusions: Physical education class participation and additional PA after school hours were both important for perceived health, PA, VO2max, and metabolic health in adulthood up to 70 years.