Background: Although the beneficial effects of physical activity and exercise on mental health are well known, the optimal conditions for them for benefitting mental health are still unclear. Engaging in exercise with others might have more desirable effects on mental health than engaging in exercise alone. This study examined the associations between exercising alone, exercising with others, and mental health among middle-aged and older adults. Methods: Baseline and 1-year follow-up surveys were conducted with 129 individuals. Time spent exercising alone or with others was measured using a 7-day diary survey. Total physical activity was objectively measured using an accelerometer. Mental well-being was assessed using the simplified Japanese version of the World Health Organization Five Well-Being Index, and mental distress was assessed using the Japanese version of the Kessler Distress Scale (K6). Results: Cross-lagged and simultaneous effects models revealed that exercising with others positively influenced mental well-being. Exercising alone and total physical activity did not significantly influence mental well-being. Neither total physical activity, exercising alone, nor exercising with others was significantly associated with mental distress. Conclusion: Engaging in exercise with others could be effective in improving mental well-being relative to engaging in exercise alone.