This study aimed to investigate if the impact of both recent and long-term physical activity on age-related cognitive decline would be modified by sex. One-hundred thirty-five men (N = 67) and women (N = 68) aged 18 to 80 years completed the Modifiable Activity Questionnaire and the Historical Leisure Activity Questionnaire. A composite score of cognitive functions was computed from five experimental tasks. Hierarchical regression analyses performed to test the moderating effect of recent physical activity on age-cognition relationship had not revealed significant result regardless of sex. Conversely, past long-term physical activity was found to slow down the age-related cognitive decline among women (β = 0.22, p = .03), but not men. The findings support a lifecourse approach in identifying determinants of cognitive aging and the importance of taking into account the moderating role of sex. This article presented potential explanations for these moderators and future avenues to explore.