This study cross-sectionally examines the relations of sitting and physical activity (PA) with cognitive impairment in community-dwelling adults aged 55–87 years (n = 3,780). Multivariable logistic regression assessed independent and joint relations of sitting and PA with Montreal Cognitive Assessment scores adjusting for covariates. Sitting ≥75% of the time and not meeting PA guidelines were related to 60% (95% confidence interval [CI] [1.19, 2.17]) and 27% (95% CI [1.06, 1.53]) higher odds for cognitive impairment, respectively. Stratification by age showed that sitting ≥75% of the time was associated with higher cognitive impairment odds in midlife (odds ratio [OR] = 1.86; 95% CI [1.31, 2.65]), but not older adults (OR = 1.06; 95% CI [0.57, 1.95]). Joint association analysis revealed that, overall, the highest odds for cognitive impairment were in those sitting ≥75% of the time while meeting or not meeting PA guidelines (OR = 1.69, 95% CI [1.13, 2.53]; and OR = 1.66, 95% CI [1.19, 2.32], respectively). In conclusion, prolonged sitting and insufficient PA are independent risk markers for cognitive impairment.