Although running contributes to a healthy lifestyle,1 it is also associated with high injury rates.2 During a 1-hour run, each foot will hit the ground approximately 10.000 times, with a peak ground reaction force (GRF) equivalent to 2.5 to 2.8 times body weight (BW).3 These forces are fundamental to the dynamics of running, determining the center of mass acceleration of the runner, and thus directly influencing running direction and speed.
The quantification of GRF is an important aspect of running biomechanics research and is frequently studied.4 GRF, or derivatives, such as loading rate, can be linked to injury risk.5–7 However, its exact relation to injuries remains a topic of debate as the evidence for GRF, and derivatives, as a determinant of injuries remains inconsistent.8 Despite this, GRF continues to be widely used in biomechanics research due to its ability to quantify external load in a relatively accessible way. Additionally, using an inverse dynamics approach, combined with musculoskeletal modeling, GRF can be combined with a kinematic measurement giving insights into structure-specific load, such as tibial bone load, Achilles tendon force, or joint contact force.9 It is believed that measuring these structure-specific loads in relation to injuries is crucial to get a better understanding of the mechanisms causing running injuries.10 Beyond increasing the understanding of mechanisms behind running injuries, GRF plays a role in assessing running performance,11 and investigating the effects of footwear on running performance.12
Traditionally, GRF is measured in a gait laboratory on force plates embedded in the floor or treadmill. Therefore accurately measuring and analyzing GRF in real-world conditions, outside of the controlled environment of a laboratory, remains a challenge. In recent years, numerous studies focused on GRF estimation using inertial sensors or measurement units (IMUs).4,13,14 IMUs are lightweight and portable, making it possible to move outside of the gait laboratory and estimate forces in the runners’ environment. However, most existing research on GRF estimation with IMUs has relied on treadmill-based protocols or indoor runway analyses, which, while valuable, do not fully replicate the variable conditions encountered during outdoor running.15
So far, only 2 studies have been published where GRF waveforms were estimated outside the laboratory using IMUs. Donahue and Hahn16,17 instrumented runners with IMUs and a force-sensing insole. A neural network was trained to estimate the GRF as measured by the force-sensing insoles, using data from an unconstrained environment. The performance achieved outside is comparable to other work if running speed was controlled.18–21 Note that Donahue and Hahn16,17 use insoles as a reference, while it is shown that these insoles do not perfectly represent forces as measured by a force plate,22 which is considered the gold standard in GRF measurement. This highlights the challenge of validating GRF estimations in an outdoor environment.
The repeatability of algorithms to measure biomechanical parameters using IMUs is of great importance for biomechanical analyses.23 High repeatability ensures consistent reproducible results under the same conditions. To get a better understanding of the etiology of running injuries, it is important to measure the biomechanical load over a longer time period, 10,24 stressing the importance of consistent measurements in the practical setting.25 So far, studies have only focused on measurements during a single visit. Therefore, the repeatability of GRF estimation algorithms is unknown.
Several factors contribute to variability in gait biomechanics and GRF both within individual sessions and across different days. Within-session variability arises from intended (change in running speed26 or stride frequency) and unintended factors (natural variability, fatigue levels, terrain, running surface), all influencing the GRF readings. Between-day variability is influenced by broader factors, such as variations in the runner’s physiological state, muscle soreness or recovery status, and environmental conditions, like weather, or surface stiffness. Finally, the positioning and attachment of the IMUs can vary slightly from session to session, potentially leading to differences in GRF estimations. In the absence of these discussed factors, an intraclass correlation coefficient (ICC) of .95 to .97 (nongraded) or .93 (10% gradient) was found between 2 days, indicating the natural day-to-day variability, in a highly controlled environment.22,27
It is known that GRF can be measured with excellent repeatability in the gait laboratory.23,27 However, with placing markers or sensors, a human factor comes into play, causing a source of errors.27 It is shown that IMU placement affects the measured sensor data28 and estimated GRF.29
This study aims to investigate the repeatability of an algorithm for the estimation of vertical GRF during running using IMUs. A previously developed algorithm using vertical acceleration of the lower legs and pelvis will be used to estimate the GRF.14 This previously developed algorithm is validated at different stride frequencies and running speeds, resulting in a root mean square error (RMSE) of .18 BW. To do this in a truly ecologically valid environment, runners ran, self-paced, outside on an athletics track on 3 different days. Data from 3 different sessions will be compared to evaluate the day-to-day repeatability. It is hypothesized that the GRF estimation algorithm will demonstrate good repeatability across the different sessions, with an ICC above .75.
Methods
Participants
For this study, 14 participants were recruited. To be eligible for participation, participants needed to have met the following criteria in the 6 months prior to inclusion: (1) they ran a minimum of 20 km/wk, (2) they ran at least twice a week, and (3) they had not suffered any significant injuries related to running. A significant running-related injury was described as an injury that persisted for more than 10 days, and forced a runner to reduce, or miss, scheduled runs due to the injury. Recruitment took place at local athletic and triathlon clubs. The University of Twente’s ethical committee approved the experimental protocol (application number 230.439), and all participants provided written informed consent before taking part in the study.
Measurement Setup
During each visit, participants were equipped with a Garmin Forerunner 225 GPS watch (Garmin Ltd). Participants were allowed to use their own Garmin watch if the release date of the watch was after 2018. Additionally, 7 Movella DOT IMUs (Movella) were placed at the (proximal) tibias and pelvis (Figure 1). As part of a larger study, more sensors were placed (on the upper legs, sternum and on the wrist underneath the strap of the GPS watch). Sensors were placed such that the x-axis of the sensor aligned with the global vertical axis if the participant was in neutral pose. Sensor placement was done by 2 researchers. Before the measurement, the internal clocks of the IMUs were synchronized and a magnetic field mapping was performed to calibrate the magnetometer, both in accordance with the manufacturer’s instructions.30 The IMUs were fixated using double-sided tape and covered with a piece of Fixomull stretch tape. It was made sure that the watch had locked onto a satellite signal before the start of the run. Watch and heart rate data (in beats per minute) were collected at 1 Hz, IMU data at 120 Hz. All data were recorded locally on sensors.
—Sensor placement on the runner. With inertial measurement units on the lower legs and pelvis, and a GPS watch on the preferred wrist.
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
All measurements started with a static trial, where the participants stood in a neutral pose (N-pose) for 10 seconds. To achieve temporal synchronization between the GPS watch data and the IMUs, the participants jumped prior to running and started the watch at the moment of touchdown.
Running Protocol
Participants were asked to visit the University of Twente athletics track (400 m, tartan running surface) for 3 visits, referred to as run 1, run 2, and run 3 (R1, R2, and R3). These 3 visits were planned over 3 weeks. Measurements were rescheduled if participants were not available, feeling ill, or if the athletics track was too slippery because of icy and snowy weather conditions.
During the first visit, participants were asked for age, gender, height, running participation, running experience, recent 5-km or 10-km race result if available, and the pace during an endurance run. The body mass of the participants was measured in their running outfits, including sensors. Body mass was assumed to be constant over the 3 sessions.
During each visit, participants ran self-paced for 4 laps on the 400-m athletics track in a counterclockwise direction at the pace they usually perform on an endurance run. This endurance pace was noted after the first visit. During the subsequent visits, participants were informed of the target running pace beforehand to ensure consistency in running speed across all 3 sessions. Additionally, the corresponding times per 200 and 400 m were given before the session for the warm-up. After these 4 laps, the participants continued another protocol as part of a larger study.
Data Processing
Data from the Garmin watch were exported as .TCX file using Garmin Connect31 website. Speed and stride frequency data were used in this study.
The sensor acceleration along the axis aligned with the tibia was inspected to find the acceleration peak caused by the jump at the moment of starting the watch. This peak was used for the temporal synchronization between the GPS watch and IMUs.
Each sample of data was classified as stance phase or flight phase using the estimated GRF data. To achieve this, initially, all data > 0 N were classified as stance. Transitions from flight phase to stance phase were dismissed if there were fewer than 4 consecutive samples above 0 N. To ensure the analysis focused solely on stance phases during continuous running, any stance phases longer than 0.45 seconds (as running ground contact times are typically shorter [0.25 s]32) and the 2 stance phases immediately before and after such instances were excluded from the analysis. All GRF waveforms were visually inspected, and none were manually removed.
To differentiate between GRF waveforms from the left and right leg during ground contact, the mediolateral axis of the gyroscope attached to the lower leg was used to identify the swinging leg. The leg with the highest magnitude along the mediolateral axis during the final samples of the ground contact phase was classified as the swinging leg, while the opposing leg was identified as the stance leg during that phase. Each stance phase was time-normalized and resampled to 100 samples to allow for comparison between different stance phases.
After 2 minutes of running, 400 steps per participant per leg were extracted for further analysis. Corresponding watch data were used to determine variability in running speed and stride frequency.
Evaluation
Per stride, the peak value, kurtosis, and skewness were calculated for the GRF waveform. Mean running speed, mean stride frequency, mean peak GRF, and the mean GRF waveforms were calculated for each participant, per session. Per participant, the absolute differences between each session were calculated for running speed, stride frequency, and peak GRF. Per participant, the root mean square difference (RMSD), relative RMSD (rRMSD, calculated as
Statistics
Kolmogorov–Smirnov tests were performed to compare differences in the distribution of running speed, as measured at 1 Hz, and the estimated GRF peaks between the sessions per participant. To assess the repeatability of the GRF estimation, the ICC (2, k) with 95% confidence intervals were calculated for the kurtosis, skewness, and detected GRF peaks.22 Values less than .5 are indicative of poor reliability, values between .5 and .75 indicate moderate reliability, values between .75 and .9 indicate good reliability, and values greater than .90 indicate excellent reliability.34
To assess the contribution of session variability to the estimation of the GRF peaks, a series of 4 linear mixed models (LMM) was applied. These models allow for the inclusion of both fixed and random effects, capturing subject-specific variability and session effects. Model 1 only included fixed and random intercepts for participants to account for intersubject variability. Model 2 included running speed as both a fixed effect and a random effect to account for intersubject variability in response to changes in speed. Model 3 is similar to model 2, however, the effect of stride frequency is included with a random effect for the variability in slope per participant. In model 4, random intercepts for sessions, nested within participants, were added to model 3 to assess the impact of repeated measurements across different sessions. Mathematically, the models are described as:
Model 1:
Model 2:
Model 3:
Model 4:
where
LMM models are well-suited to account for the fixed effects of running speed on GRF as well as the random variability introduced by both participants and repeated sessions, enabling quantification of session-to-session variability. Model fit was assessed using the Akaike Information Criterion, Bayesian Information Criterion, RMSE, and R2. The variance components for the random effects (participant and session) were also extracted to evaluate the proportion of variability attributable to each source. Finally, the ICC (2, k, 2-way random effects, absolute agreements, average measurements) was calculated to quantify the consistency of GRF peak and shape (kurtosis and skewness) estimates across sessions. The LMMs were fitted in R, all other data processing was done in Python.
Results
From the initial 14 included participants, 2 participants were injured between the first and second sessions (not during measurements) and were excluded from the results. Data from 12 participants (7 males, 5 females, 27.9 [7.6] y old, 1.83 [0.12] m, 72.1 [9.0] kg) were included in this study. The population was well trained, with 10.1 (6.9) years of running experience and 39 (19) km/wk on average. Participants were renumbered to 1 to 12.
The algorithm for GRF estimation shows a repeatable shape across different sessions. For example, S02 (top Figure 2) has a larger passive GRF peak (at midstance) at the right leg, compared to the left leg for all sessions. The repeatability in shape of GRF waveforms is reflected in the ICC values of .92 (95% confidence interval, .79–.97) and .94 (95% confidence interval, .86–.98) for skewness and kurtosis, respectively. ICC of .86 (95% confidence interval, .62–.96) was found for the between-day reliability over 3 days for the maximum GRF. Repeatability values per participant are shown in Table 1. Participants 02 and 03 demonstrated the highest repeatability as represented by the lowest RMSD (average < 0.05 BW for S02 and S03; Table 1, Figure 2). Participants 04 and 09 showed the highest RMSD (0.16 and 0.13 BW; Figure 3).
—Estimated GRF waveform for S02 (top) and S03 (bottom) for left and right leg. These participants had the lowest RMSD. R1, R2, and R3 represent the runs on the 3 different days. Shading represents the standard deviation. BW indicates body weight; GRF, ground reaction force; RMSD, root mean squared difference.
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
Average Differences Over the 3 Sessions per Participant
| Participant | Pearson r | RMSD (× BW) | rRMSD, % | Rel peak difference, % | Abs peak difference (× BW) | Abs speed difference, m/s | Abs stride frequency difference, strides/min |
|---|---|---|---|---|---|---|---|
| 01 | .999 | .06 | 2.5 | 2.4 | 0.06 | 0.1 | 0.3 |
| 02 | .999 | .05 | 1.8 | 0.9 | 0.02 | 0.1 | 1.2 |
| 03 | .999 | .05 | 1.9 | 0.9 | 0.02 | 0.1 | 1.3 |
| 04 | .997 | .16 | 6.3 | 9.7 | 0.25 | 0.1 | 0.5 |
| 05 | .997 | .10 | 4.4 | 7.0 | 0.16 | 0.0 | 0.3 |
| 06 | .998 | .06 | 2.5 | 2.6 | 0.06 | 0.1 | 3.2 |
| 07 | .998 | .06 | 2.5 | 2.2 | 0.05 | 0.0 | 1.8 |
| 08 | .994 | .10 | 4.4 | 4.0 | 0.09 | 0.2 | 1.0 |
| 09 | .998 | .13 | 5.1 | 7.1 | 0.17 | 0.1 | 0.6 |
| 10 | .998 | .07 | 2.9 | 1.5 | 0.04 | 0.2 | 1.0 |
| 11 | .997 | .06 | 2.6 | 1.2 | 0.02 | 0.1 | 1.0 |
| 12 | .997 | .10 | 3.9 | 4.0 | 0.10 | 0.2 | 0.9 |
| Mean | .998 | .08 | 3.4 | 3.6 | 0.09 | 0.1 | 1.1 |
Abbreviations: Abs, absolute; BW, bodyweight; RMSD, root mean square difference; rRMSD, relative RMSD; Rel, relative.
—Estimated GRF waveform for S04 (top) and S09 (bottom) for left and right leg. These participants had the highest RMSD. R1, R2, and R3 represent the runs on the 3 different days. Shading represents the standard deviation. BW indicates body weight; GRF, ground reaction force; RMSD, root mean squared difference.
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
Average running speed over all sessions was 3.2 (0.2) m/s (11.5 [0.8] km/h). The mean difference in running speed between the sessions was 0.1 (0.1) m/s (0.4 [0.3] km/h), which is equivalent to an average difference of 3.1% (2.6%) (Figure 4); however, the distribution of running speed was significantly different between all sessions per participant (alpha < .05). Stride frequency was 84.2 strides/min on average with a SD of 3.4 strides/min.
—Distribution of speed by participant and session. All differences between the sessions were significant. R1, R2, and R3 represent the runs on the 3 different days. The average values and SDs can be found in Supplementary Materials (see Supplemental Table A1 [available online]).
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
The intercept-only LMM (model 1) had an R2 of .71, but when speed was added (model 2), R2 increased to .84, and all other performance indicators showed an improvement. The fixed effects of speed (Table 2) show a positive relation with GRF, indicating that a higher speed results in a higher GRF (0.21 times BW increase per increase in 1 m/s). Model performance increased with the addition of stride frequency (model 3, R2 = .90), where stride frequency shows a negative relation with GRF (Table 3), meaning that a higher stride frequency results in a lower GRF. In model 4, the effect of session was added, resulting in an improved model performance (R2 = .93), however, the improvement is more prominent in the decrease in RMSE from .071 to .047 BW (Table 3). The model parameters of model 4 indicate that the random effect of participant on the intercept is much higher than the random effect of session on the intercept (0.2964 vs 0.0075; Table 3).
Performance Indicators for the Linear Mixed Models
| Performance indicators | Model 1: fixed and random intercept per participant | Model 2: model 1 + speed effects with fixed and random slope per participant | Model 3: model 2 + stride frequency effects with fixed and random slope | Model 4: model 3 + random intercept per session |
|---|---|---|---|---|
| AIC | −29,637.48 | −31,702.09 | −34,011.60 | −45,584.27 |
| BIC | −29,614.84 | −31,656.80 | −33,936.13 | −45,501.25 |
| R2 | .7090 | .8422 | .9045 | .9324 |
| RMSE | .0836 | .0774 | .0710 | .0468 |
Abbreviations: AIC, Akaike Information Criterion; BIC, Bayesian Information CriterionRMSE, root mean squared error.
Model Parameters and Correlation Between Parameters of the Linear Mixed Models
| Model parameters | Model 1: fixed and random intercept per participant | Model 2: model 1 + speed effects with fixed and random slope per participant | Model 3: model 2 + stride frequency effects with fixed and random slope | Model 4: model 3 + random intercept per session |
|---|---|---|---|---|
| Fixed effect intercept | 2.4030 | 1.7258 | 3.0687 | 2.1149 |
| Fixed effects speed | — | 0.2083 | 0.1818 | 0.0112 |
| Fixed effects stride frequency | — | — | −0.0152 | 0.0029 |
| Variance random effect participant (intercept) | 0.0171 | 0.9490 | 3.7761 | 0.2964 |
| Variance random effect participant (speed) | — | 0.1028 | 0.0934 | 0.0081 |
| Variance random effect participant (stride frequency) | — | — | 0.0008 | 0.00003 |
| Variance random effect participant: session (intercept) | — | — | — | 0.0075 |
| Correlation (intercept, speed) | — | −.988 | .267 | −.546 |
| Correlation (intercept, stride frequency) | — | — | −.920 | −.738 |
| Correlation (speed, stride frequency) | — | — | −.618 | −.111 |
The evaluation metrics for repeatability of the estimation of GRF are shown per participant (Figure 5). Note that, for example, for S04, it is seen that there are 2 larger bars for RMSD. This does mean that one session is deviating from the other 2, in this particular case, the R2 session. This is shown in Figure 3, where the orange waveform for R2 deviates from the other sessions for both the left and right leg.
—Comparison between root mean squared difference, absolute peak difference, speed difference, and stride frequency difference by participant and session. R1, R2, and R3 represent the runs on the 3 different days. The exact values can be found in Supplementary Materials (see Supplemental Table A2 [available online]). BW indicates body weight.
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
All Pearson correlation coefficients between the sessions per participant are greater than .99 and are .997 on average (Table 1, Figure 5). RMSD between the sessions per participant is .08 BW on average, and the average peak error was .09 BW (3.7%; Table 1). The distribution of GRF peaks differed significantly between all sessions per participant (alpha < .05; Figure 6).
—Distribution of the estimated GRF peaks by participants and session, normalized to BW. R1, R2, and R3 represent the runs on the 3 different days. The average values and SDs can be found in Supplementary Materials (see Supplemental Table A3 [available online]). BW indicates body weight; GRF, ground reaction force.
Citation: Journal of Applied Biomechanics 41, 2; 10.1123/jab.2024-0126
Discussion
Twelve participants ran 3 sessions at the athletics track with the main goal of evaluating the repeatability of a vertical GRF estimation algorithm,14 using 3 IMUs, over 3 different days. An ICC of .86 was found for the GRF peaks, indicating good reliability,34 aligning with the hypothesis that the ICC would exceed .75. Differences between the sessions were small with an average Pearson correlation coefficient of .99 and RMSD of .08 BW. To put these values into perspective, these differences are within the model accuracy of a Pearson correlation coefficient of .97 and RMSE of .19 BW. An average absolute peak difference of 3.70% is found, while the model accuracy is 4.21%.33 Differences between participants were captured and were consistent over the different sessions, as indicated by the high ICC for skewness and kurtosis (ICC > .92).
The algorithm’s sensitivity to speed and stride frequency in estimating the GRF peaks is shown using the LMM, where a positive relation was shown with speed and a negative relation with increasing stride frequency, in line with the literature.35,36 Model 3, where GRF peak was estimated as an intercept with effect of speed and stride frequency, all with random (participant-specific) effects, shows an R2 of .90 (Table 3). If a random effect of session (model 4) was added to the model, it increased to .93.
The addition of session to the model resulted in a decrease of RMSE from .07 to .05 BW. This means that session effects, either resulting from sensor placement, or natural day-to-day variability, has an effect on the GRF peaks. The random effect variance of .30 for participants, compared with .01 for sessions (Table 3), indicates that variability between participants is greater than variability between sessions.
In model 4 of the LMM series, the fixed effects of speed and stride frequency decreased from 0.18 to 0.01 and −0.2 to 0.003, respectively (Table 3), compared with model 3. As the LMM complexity increased, so did its flexibility, allowing for adjustments in the random effect for session to account for any differences in speed and stride frequency across sessions.
Although a lot of research is performed on estimating GRF while running with a set up suitable for outdoor running,16,19,21,33,37–40 this is the first study that examined the day-to-day repeatability of such an algorithm. Day-to-day repeatability is a crucial aspect of validating the practical applicability of GRF estimation algorithms for outdoor running scenarios. High day-to-day repeatability, while still maintaining sensitivity toward biomechanical changes, ensures that the algorithm can provide consistent results across different sessions, which is essential for monitoring changes in running biomechanics over time.27 The model used in this study is therefore assumed to be applicable with good repeatability in daily practice.
In the past studies using IMUs to estimate GRF were mainly conducted in a laboratory environment, indoors, on a treadmill, at prescribed speeds and/or over small distances.15 During this study, runners were equipped with sensors, received instructions on running pace, and then started running on the athletics track, closely replicating natural running conditions. Although this causes fluctuations in running speed, it represents real-world running.
Even if a measurement system or estimation algorithm is perfect, still some day-to-day variability is expected because of the variability in the human body. This variability in running kinematics and kinetics can be explained by day-to-day changes in muscle flexibility because of effects such as physical development, aging, body temperature, and training. Furthermore, it is shown that fatigue causes changes in kinematics during running.41 It can be argued that preexercise fatigue would cause similar changes in kinematics, causing day-to-day variability in running. It is even shown in 2 studies that the peak GRF of the same participant, at the same speed while running on a treadmill has an ICC of .95 to .97 (nongraded) or .93 (10% gradient) between 2 days, meaning there is natural day-to-day variability, even in a highly controlled environment.22,27 The ICC of .86, found in this study, is very close to the natural day-to-day variability, meaning that the algorithm to estimate GRF is suitable for daily monitoring.
During the running sessions in this study, participants were instructed to run with a certain running speed but had to control this themselves. Using lap times and pace output from a GPS watch, they were able to adjust accordingly. Despite this, speed distribution between sessions always differed significantly, however, the high power of the test means that even minor variations appear significantly different. On average, a difference in speed was seen of 0.1 m/s, but outliers up to 0.3 m/s between sessions were observed (Figure 4). It is known and shown with the LMM that GRF increases with running speed,26 influencing the measured repeatability in this study. These fluctuations in running speed could explain some of the day-to-day differences found in this study. GPS is seen as reliable for measuring constant running speed.42
Furthermore, the attachment method of the sensor to the body affects the measured accelerations.43 In our study, sensors were placed with double-sided tape, secured with an additional piece of tape on top. However, moisture from either sweat or rain conditions could cause a slight detachment of sensors affecting the data.
It is known that the IMU itself44 and the orientation filter within the IMU can affect the data and thus GRF estimates. Often an orientation filter is used to get information on sensor orientation when using IMUs for gait analyses or to subtract the gravitational acceleration to obtain the linear acceleration. Algorithms to estimate GRF can be dependent on this orientation estimation.16,17,33 In case of magnetic field distortions, errors in orientation estimation can arise.45 The algorithm used in this study only relies on an estimate of inclination, as only the vertical acceleration is used for the estimation, which is more accurate than heading angle.46
Another technical factor that could influence the reliability when measuring with IMUs is the sensor-to-segment calibration. The goal of this calibration is to obtain the mapping of the sensor frame to the frame of a specific body segment. This can be done using predefined postures or movements.47 This means that the execution of the calibration can influence the reliability of the obtained results.48 For example, it is shown that misalignment between the sensor and the body segment influences the estimation of the knee joint angle.49 Note that this factor is only of relevance if an algorithm for GRF estimation is dependent on this calibration, for example, as done by Verheul et al38 and Wille et al50 Ideally, algorithms for daily use in the runners’ environment do not require a sensor-to-segment calibration or orientation estimation, and are independent of sensor placement. The algorithm tested in this study does not require sensor-to-segment calibration and the same orientation filter was used every session. This means that the variability between the sessions, apart from the natural variability, is most likely the result of sensor placement and attachment.
Besides technological factors causing intrasubject differences between days, there are important other factors that contribute to day-to-day variability, especially in an outdoor setting. As measurements were performed during the winter, a range of temperature between −10 and 10 °C was seen. With such a temperature difference, it is known that shoe material properties change51,52 and alter the shock attenuation of the shoe.53 Besides the effect of the weather on the shoes, also properties of the track itself are probably altered with humidity and temperature.54 Additionally, on certain measurement days, the track was wet caused by rain, decreasing the friction between the shoe and the track, potentially resulting in different running biomechanics. Due to these external factors, ICCs measured outdoors are always expected to be lower than ICCs measured indoors.
Conducting biomechanical research in outdoor running environments presents challenges, particularly the absence of a gold standard for GRF measurements. Unlike controlled laboratory settings, where force plates or instrumented treadmills provide the “true data,” outdoor conditions are inherently variable and lack comparable measurement systems. This limitation complicates the validation of GRF estimation methods, as it becomes challenging to reason whether observed between-session variability arises from true biomechanical changes (eg, due to fatigue, injury, running surface, speed, or technique adaptations) or from methodological inaccuracies. While natural running environments offer ecological validity, the absence of a gold standard requires careful interpretation of findings and additional analyses, such as done in our study with the LMM. High repeatability, for example, may suggest consistent estimations but could also indicate a lack of sensitivity to subtle biomechanical changes. On the other hand, high variability could reflect either genuine biomechanical differences or methodological limitations. Future research combining indoor and outdoor measurements may help address this gap, enabling more robust validations of sensor technology on runners. Additionally, repeatability analyses during outdoor running should always be accompanied by an analysis of sensitivity toward biomechanical gait changes.
The level of repeatability required for meaningful GRF monitoring depends on the specific application and context in which it is used. In biomechanics research, particularly in studies focusing on injury prevention, fatigue monitoring, or performance assessment, there are no fixed thresholds on repeatability level. Our findings highlight the challenges of achieving high repeatability in natural running environments, where variability in speed, terrain, and sensor placement can influence GRF estimations.
This is the first study to evaluate day-to-day repeatability of an algorithm in an outdoor setting for estimation of GRF and represents a step forward in the field of running biomechanics. The evaluated algorithm demonstrates a good level of repeatability, with an ICC of .86 for peak GRF, mean RMSD of .08 BW, and Pearson correlation coefficients above .99, compared over 3 days. The GRF waveform showed that individual movement patterns and asymmetries between the left and right leg were preserved over the different sessions. This indicates that the algorithm can reliably capture individual running patterns and maintain these patterns across 3 different sessions. The study is performed in an ecologically valid environment, which better reflects the conditions under which athletes and recreational runners typically run. This is the first study evaluating the day-to-day reliability of a GRF estimation algorithm. The outcome shows the potential to estimate GRF in the outdoor environment during running.
Acknowledgments
The authors would like to thank Lieke Kamperman for her assistance in performing the measurements. Data statement: The data that support the findings of this study are available from the corresponding author, Scheltinga upon reasonable request. Funding details: This work is part of the multidisciplinary research project Perfect Fit, which is supported by several funders organized by the Netherlands Organization for Scientific Research (NWO), program Commit2Data—Big Data & Health (project number 628.011.211). Besides NWO, the funders include the Netherlands Organization for Health Research and Development (ZonMw), Hartstichting, the Ministry of Health, Welfare and Sport (VWS), Health Holland, and the Netherlands eScience Center.
References
- 1.↑
Lee DC, Pate RR, Lavie CJ, Sui X, Church TS, Blair SN. Leisure-time running reduces all-cause and cardiovascular mortality risk. J Am Coll Cardiol. 2014;64(5):472–481. doi:
- 2.↑
Van Gent RN, Siem D, Van Middelkoop M, Van Os AG, Bierma-Zeinstra SMA, Koes BW. Incidence and determinants of lower extremity running injuries in long distance runners: a systematic review. Br J Sports Med. 2007;41(8):469–480. doi:
- 3.↑
Cavanagh PR, Lafortune MA. Ground reaction forces in distance running. J Biomech. 1980;13(5):397–406. doi:
- 4.↑
Kiernan D, Ng B, Hawkins DA. Acceleration-based estimation of vertical ground reaction forces during running: a comparison of methods across running speeds, surfaces, and foot strike patterns. Sensors. 2023;23(21):8719. doi:
- 5.↑
Kiernan D, Hawkins DA, Manoukian MAC, et al. Accelerometer-based prediction of running injury in National Collegiate Athletic Association track athletes. J Biomech. 2018;73:201–209. doi:
- 6.
Johnson CD, Tenforde AS, Outerleys J, Reilly J, Davis IS. Impact-related ground reaction forces are more strongly associated with some running injuries than others. Am J Sports Med. 2020;48(12):3072–3080. doi:
- 7.↑
Ceyssens L, Vanelderen R, Barton C, Malliaras P, Dingenen B. Biomechanical risk factors associated with running-related injuries : a systematic review. Sports Med. 2019;49(7):1095–1115. doi:
- 8.↑
Matijevich ES, Branscombe LM, Scott LR, Zelik KE. Ground reaction force metrics are not strongly correlated with tibial bone load when running across speeds and slopes: implications for science, sport and wearable tech. PLoS One. 2019;14(1):10000. doi:
- 9.↑
Scott SH, Winter DA. Internal forces of chronic running injury sites. Med Sci Sports Exerc. 1990;22(3):357–369. https://www.ncbi.nlm.nih.gov/pubmed/2381304
- 10.↑
Bertelsen ML, Hulme A, Petersen J, et al. A framework for the etiology of running-related injuries. Scand J Med Sci Sport. 2017;27(11):1170–1180. doi:
- 11.↑
Hunter JP, Marshall RN, McNair PJ. Relationships between ground reaction force impulse and kinematics of sprint-running acceleration. J Appl Biomech. 2005;21(1):31–43. doi:
- 12.↑
Healey LA, Hoogkamer W. Longitudinal bending stiffness does not affect running economy in Nike Vaporfly Shoes. J Sport Heal Sci. 2022;11(3):285–292. doi:
- 13.↑
Ancillao A, Tedesco S, Barton J, O’flynn B. Indirect measurement of ground reaction forces and moments by means of wearable inertial sensors: a systematic review. Sensors. 2018;18(8):564. doi:
- 14.↑
Scheltinga BL, Usta H, Reenalda J, Buurke JH. Estimating vertical ground reaction force during running with 3 inertial measurement units. J Biomed Eng Biosci. 2022;9:6. doi:
- 15.↑
Benson LC, Räisänen AM, Clermont CA, Ferber R. Is this the real life, or is this just laboratory? A scoping review of IMU-based running gait analysis. Sensors. 2022;22(5):1722. doi:
- 16.↑
Donahue SR, Hahn ME. Estimation of ground reaction force waveforms during fixed pace running outside the laboratory. Front Sport Act Living. 2023;5:186. doi:
- 17.↑
Donahue SR, Hahn ME. Estimation of gait events and kinetic waveforms with wearable sensors and machine learning when running in an unconstrained environment. Sci Rep. 2023;13(1):9314. doi:
- 18.↑
Dorschky E, Nitschke M, Martindale CF, van den Bogert AJ, Koelewijn AD, Eskofier BM. CNN-based estimation of sagittal plane walking and running biomechanics from measured and simulated inertial sensor data. Front Bioeng Biotechnol. 2020;8:604. doi:
- 19.↑
Wouda FJ, Giuberti M, Bellusci G, et al. Estimation of vertical ground reaction forces and sagittal knee kinematics during running using three inertial sensors. Front Physiol. 2018;9:218. doi:
- 20.
Ohtaki Y, Sagawa K, Inooka H. A method for gait analysis in a daily living environment by body-mounted instruments. JSME Int J Ser C. 2001;44(4):1125–1132. doi:
- 21.↑
Alcantara RS, Edwards WB, Millet GY, Grabowski AM. Predicting continuous ground reaction forces from accelerometers during uphill and downhill running: a recurrent neural network solution. PeerJ Comput Sci. 2022;10:e12752. doi:
- 22.↑
Renner K, Williams D, Queen R. The reliability and validity of the Loadsol® under various walking and running conditions. Sensors. 2019;19(2):265. doi:
- 23.↑
Kadaba MP, Ramakrishnan HK, Wootten ME, Gainey J, Gorton G, Cochran GVB. Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait. J Orthop Res. 1989;7(6):849–860. doi:
- 24.↑
Yeung SS, Yeung EW, Gillespie LD. Interventions for preventing lower limb soft-tissue running injuries. Cochrane Database Syst Rev. 2011;10(7):CD001256. doi:
- 25.↑
Hughes GTG, Camomilla V, Vanwanseele B, Harrison AJ, Fong DTP, Bradshaw EJ. Novel technology in sports biomechanics: some words of caution. Sports Biomech. 2021;10:453. doi:
- 26.↑
Hamill J, Bates BT, Knutzen KM, Sawhill JA. Variations in ground reaction force parameters at different running speeds. Hum Mov Sci. 1983;2(1–2):47–56. doi:
- 27.↑
Ferber R, McClay Davis I, Williams DS, Laughton C. A comparison of within- and between-day reliability of discrete 3D lower extremity variables in runners. J Orthop Res. 2002;20(6):1139–1145. doi:
- 28.↑
Van Middelaar RP, Zhang J, Veltink PH, Reenalda J. 3D tibial acceleration and consideration of 3D angular motion using IMUs on peak tibial acceleration and impulse in running. Med Sci Sports Exerc. 2023;55(12):2253–2262. doi:
- 29.↑
Kiernan D, Katzman ZD, Hawkins DA, Christiansen BA. A 0.05 m change in inertial measurement unit placement alters time and frequency domain metrics during running. Sensors. 2024;24(2):656. doi:
- 30.↑
Movella. Xsens DOT User Manual. 2023:1–34. Accessed February 23, 2024. https://www.movella.com/hubfs/Xsens%25DOT%25User%25Manual.pdf
- 32.↑
Moore IS, Ashford KJ, Cross C, Hope J, Jones HSR, Mccarthy-ryan M. Humans optimize ground contact time and leg stiffness to minimize the metabolic cost of running. Front Sport Act Living. 2019;1:53. doi:
- 33.↑
Scheltinga BL, Kok JN, Buurke JH, Reenalda J. Estimating 3D ground reaction forces in running using three inertial measurement units. Front Sport Act Living. 2023;5:466. doi:
- 34.↑
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155. doi:
- 35.↑
Jiang X, Bíró I, Sárosi J, Fang Y, Gu Y. Comparison of ground reaction forces as running speed increases between male and female runners. Front Bioeng Biotechnol. 2024;12:1378284. doi:
- 36.↑
Lieberman DE, Warrener AG, Wang J, Castillo ER. Effects of stride frequency and foot position at landing on braking force, hip torque, impact peak force and the metabolic cost of running in humans. J Exp Biol. 2015;218(21):3406–3414. doi:
- 37.↑
LeBlanc B, Hernandez EM, McGinnis RS, Gurchiek RD. Continuous estimation of ground reaction force during long distance running within a fatigue monitoring framework: a Kalman filter-based model-data fusion approach. J Biomech. 2021;115:110130. doi:
- 38.↑
Verheul J, Gregson W, Lisboa P, Vanrenterghem J, Robinson MA. Whole-body biomechanical load in running-based sports: the validity of estimating ground reaction forces from segmental accelerations. J Sci Med Sport. 2019;22(6):716–722. doi:
- 39.
Charry E, Hu W, Umer M, Ronchi A, Taylor S. Study on estimation of peak Ground Reaction Forces using tibial accelerations in running. In: Palaniswami M, Leckie C, Kanhere S, Gubbi J, eds. Proceedings of the 2013 IEEE 8th International Conference on Intelligent Sensors, Sensor Networks and Information Processing: Sensing the Future. Vol 1. ISSNIP; 2013:288–293. doi:
- 40.↑
Neugebauer JM, Collins KH, Hawkins DA. Ground reaction force estimates from ActiGraph GT3X+ hip accelerations. PLoS One. 2014;9(6):9023. doi:
- 41.↑
Zandbergen MA, Marotta L, Bulthuis R, Buurke JH, Veltink PH, Reenalda J. Effects of level running-induced fatigue on running kinematics: a systematic review and meta-analysis. Gait Posture. 2023;99:60–75. doi:
- 42.↑
Huggins RA, Giersch GEW, Belval LN, et al. The validity and reliability of global positioning system units for measuring distance and velocity during linear and team sport simulated movements. J Strength Cond Res. 2020;34(11):3070–3077. doi:
- 43.↑
Sheerin KR, Reid D, Besier TF. The measurement of tibial acceleration in runners—a review of the factors that can affect tibial acceleration during running and evidence-based guidelines for its use. Gait Posture. 2019;67:12–24. doi:
- 44.↑
Gruber AH, Marotta L, McDonnell J, Reenalda J. All devices are not created equal: simultaneous data collection of three triaxial accelerometers sampling at different frequencies. Proc Inst Mech Eng Part P J Sport Eng Technol. 2022;10:517. doi:
- 45.↑
de Vries WHK, Veeger HEJ, Baten CTM, van der Helm FCT. Magnetic distortion in motion labs, implications for validating inertial magnetic sensors. Gait Posture. 2009;29(4):535–541. doi:
- 46.↑
Laidig D, Caruso M, Cereatti A, Seel T. BROAD—a benchmark for robust inertial orientation estimation. Data. 2021;6(7):72. doi:
- 47.↑
Bonnet S, Bassompierre C, Godin C, Lesecq S, Barraud A. Calibration methods for inertial and magnetic sensors. Sensors Actuators A Phys. 2009;156(2):302–311. doi:
- 48.↑
Kamstra H, Wilmes E, van der Helm FCT. Quantification of error sources with inertial measurement units in sports. Sensors. 2022;22(24):765. doi:
- 49.↑
Jiang C, Yang Y, Mao H, Yang D, Wang W. Effects of dynamic IMU-to-segment misalignment error on 3-DOF knee angle estimation in walking and running. Sensors. 2022;22(22):9009. doi:
- 50.↑
Wille CM, Lenhart RL, Wang S, Thelen DG, Heiderscheit BC. Ability of sagittal kinematic variables to estimate ground reaction forces and joint kinetics in running. J Orthop Sports Phys Ther. 2014;44(10):825–830. doi:
- 51.↑
Kinoshita H, Bates BT. The effect of environmental temperature on the properties of running shoes. J Appl Biomech. 1996;12(2):258–268. doi:
- 52.↑
Sanchez C. Effects of temperature on running shoes. RunRepeat. 2024. Accessed March 4, 2024. https://runrepeat.com/guides/effects-temperature-running-shoes
- 53.↑
Dib MY, Smith J, Bernhardt KA, Kaufman KR, Miles KA. Effect of environmental temperature on shock absorption properties of running shoes. Clin J Sport Med. 2005;15(3):172–176. doi:
- 54.↑
Andena L, Briatico-Vangosa F, Cazzoni E, Ciancio A, Mariani S, Pavan A. Modeling of shock absorption in athletics track surfaces. Sport Eng. 2015;18(1):162. doi:
