Research has found that social relationships are central to the health and well-being of an aging population. Evidence exploring the association between physical activity (PA) and sedentary behavior (SB) with social isolation and loneliness is limited. This study uses objectively measured PA and SB (ActiGraph®) and self-reported measures of loneliness (the De Jong Gierveld Loneliness Scale) and social engagement (the Lubben Social Network Scale) from the SITLESS study, a European-wide study of community-dwelling older adults. Social isolation was associated with SB where higher levels of SB were associated with an increase in the level of social isolation, controlling for age, sex, living arrangements, employment status, body mass index, educational background, marital status, and self-reported general health. In contrast, PA was not associated with social isolation, and neither SB nor PA was a statistically significant predictor of loneliness. SB may be linked to social isolation in older adults, but PA and SB are not necessarily linked to loneliness in older community-dwelling adults.