This prospective cohort study aimed to investigate the association between physical activity (PA) as measured using accelerometers, and functional improvement measured using a short physical performance battery in older patients undergoing rehabilitation. After admission to the rehabilitation hospital, patients were categorized into quartile groups based on their level of PA measured using accelerometers. The primary outcome was physical function measured using the short physical performance battery at hospital discharge. A total of 204 patients were included in the analysis. After adjusting for confounding factors, light-intensity PA (p < .001) and moderate-to-vigorous-intensity PA (p < .001) were associated with a short physical performance battery at hospital discharge. In conclusion, PA at admission is positively associated with functional improvement in older patients undergoing hospital rehabilitation.