The purpose of this study was to explore cross-sectional relationships between self-reported physical activity (PA) and personal, social, and environmental factors in community-dwelling adults aged 50 years and older. Accounting for clustering by neighborhood, generalized estimating equations were used to examine associations between selected correlates and the Physical Activity Scale for the Elderly (PASE) score while adjusting for confounders. Data for 601 participants were analyzed: 79% female, 37% married, mean age 76.8 (± 8.7) years, mean PASE score 112.6 (± 64.8). Age, living in seniors’ housing, using nursing/home care services, receiving encouragement to be active, and having benches available in the neighborhood were inversely associated with PASE. Self-efficacy, SF-12 score, PA barriers, social support, and the presence of trails showed positive associations. Several personal, social, and environmental factors associated with PA were identified. The inverse association between PA and living in seniors’ housing units should be considered when developing PA programs for older adults.