Objectives: This study aims to examine associations between changes in physical education, anxiety, cardiorespiratory fitness (CRF), and muscular fitness (MF) in adolescents. Methods: Participants were Finnish students (baseline: n = 1,147; 11.27 [0.32] years). Data were collected over 4 years (2017–2021). CRF was assessed with the 20 m shuttle run, MF using curl-up/push-up tests, and anxiety with the Physical Education State Anxiety Scale. The random intercept cross-lagged panel model, including repeated measures (within-level) and latent levels (between-level) of anxiety in PE, CRF, and MF, was used. The covariates considered were sex, body mass index, and peak height velocity. Results: CRF and anxiety increased, while MF remained stable over time. Lower fitness was associated with greater anxiety (and vice versa). Sex (female), greater body mass index, and undergoing peak height velocity were associated with greater anxiety. Conclusions: Negative experiences in PE are associated with decreased CRF and MF. Effective teaching practices are needed to optimize adolescents’ experiences in PE.