A coach’s nonverbal communication (NC) plays a central role in the construction of the coach–athlete relationship. Moreover, perceived NC and its effect on communication ability and coaching evaluation may differ according to the athletes’ demographics. This study explored the impact of perceived NC on coaching evaluation and overall communication among different genders and age groups. The study recruited 233 athletes from five high schools and seven university teams in Japan. The coaches’ NC, communication ability, and coaching evaluations were assessed. Negative and positive NC directly influenced coaching evaluation in female athletes. Specifically, negative NC directly impacted coaching evaluation in high school athletes, whereas positive NC directly influenced coaching evaluation in university athletes. Positive NC consistently influenced communication ability regardless of demographics. The findings promote talent development and team management in the coaching context.