The study goal was to examine the association between perceived neighborhood characteristics and walking in urban older adults in Brazil. A cross-sectional study including 4,027 older adults from the baseline of the Brazilian Longitudinal Study of Aging (ELSI-Brazil) was performed. Walking was measured using the International Physical Activity Questionnaire. Neighborhood characteristics were questions about physical disorder, noise pollution, safety, violence, social cohesion, services, concerns with community mobility, and pleasantness. Multinomial logistic regression was used. Concern about taking the bus, subway, or train was inversely associated with walking for men. Violence (victim of theft, robbery, or had home broken into) and social cohesion (trust in neighbors) were positively and inversely associated with walking for women, respectively. A significant interaction term between social cohesion and number of chronic diseases was observed for women. These findings demonstrate the need for sex-specific interventions and policies to increase the walking levels among older Brazilian adults.