The purpose of this study was to examine adolescents’ physical self-perceptions and their associations with physical activity using a longitudinal perspective. Utilizing data from the Physical Activity in Scottish Schoolchildren (PASS) study, changes in exercise self-efficacy, perceived competence, global self-esteem and physical self-worth were assessed among a sample of 641 Scottish adolescents from age 11–15 years. Girls reported lower levels of perceived competence, self-esteem and physical self-worth than boys at each age. Furthermore, girls’ physical self-perceptions decreased markedly over time. Among boys, only perceived competence decreased, while global self-esteem increased. Baseline physical activity was a significant predictor of later activity levels for both genders. Findings demonstrate the importance of physical self-perceptions in relation to physical activity behavior among adolescents. Among older boys, high perceived competence increased the odds of being active by 3.8 times. Among older girls, high exercise self-efficacy increased the odds of being active by 5.2 times. There is a need for early interventions which promote increased physical literacy and confidence, particularly among girls.