Evaluating the Performance of Sensor-Based Bout Detection Algorithms: The Transition Pairing Method

in Journal for the Measurement of Physical Behaviour
View More View Less
  • 1 The University of Tennessee, Knoxville
Restricted access

Purchase article

USD  $24.95

Student 1 year online subscription

USD  $39.00

1 year online subscription

USD  $52.00

Student 2 year online subscription

USD  $74.00

2 year online subscription

USD  $99.00

Background: Bout detection algorithms are used to segment data from wearable sensors, but it is challenging to assess segmentation correctness. The purpose of this study was to present and demonstrate the Transition Pairing Method (TPM), a new method for evaluating the performance of bout detection algorithms. Methods: The TPM compares predicted transitions to a criterion measure in terms of number and timing. A true positive is defined as a predicted transition that corresponds with one criterion transition in a mutually exclusive pair. The pairs are established using an extended Gale-Shapley algorithm, and the user specifies a maximum allowable within-pair time lag, above which pairs cannot be formed. Unpaired predictions and criteria are false positives and false negatives, respectively. The demonstration used raw acceleration data from 88 youth who wore ActiGraph GT9X monitors (right hip and non-dominant wrist) during simulated free-living. Youth Sojourn bout detection algorithms were applied (one for each attachment site), and the TPM was used to compare predicted bout transitions to the criterion measure (direct observation). Performance metrics were calculated for each participant, and hip-versus-wrist means were compared using paired t-tests (α = 0.05). Results: When the maximum allowable lag was 1-s, both algorithms had recall <20% (2.4% difference from one another, p < .01) and precision <10% (1.4% difference from one another, p < .001). That is, >80% of criterion transitions were undetected, and >90% of predicted transitions were false positives. Conclusion: The TPM improves on conventional analyses by providing specific information about bout detection in a standardized way that applies to any bout detection algorithm.

Hibbing, LaMunion, and Crouter are with the Department of Kinesiology, Recreation, and Sport Studies; Hilafu is with the Department of Business Analytics and Statistics; The University of Tennessee, Knoxville, Knoxville, TN, USA.

Hibbing (phibbing@vols.utk.edu) is corresponding author.

Supplementary Materials

    • Supplementary Material 1 (PDF 83 KB)
    • Supplementary Material 2 (PDF 53 KB)
    • Supplementary Material 3 (PDF 19 KB)
  • Aminikhanghahi, S., & Cook, D.J. (2017). A survey of methods for time series change point detection. Knowledge and Information Systems, 51(2), 339367. PubMed ID: 28603327 doi:10.1007/s10115-016-0987-z

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ayabe, M., Kumahara, H., Morimura, K., & Tanaka, H. (2013). Epoch length and the physical activity bout analysis: An accelerometry research issue. BMC Research Notes, 6(1), 20. PubMed ID: 23331772 doi:10.1186/1756-0500-6-20

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ellingson, L.D., Schwabacher, I.J., Kim, Y., Welk, G.J., & Cook, D.B. (2016). Validity of an integrative method for processing physical activity data. Medicine and Science in Sports and Exercise, 48(8), 16291638. PubMed ID: 27015380 doi:10.1249/MSS.0000000000000915

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Feuz, K.D., Cook, D.J., Rosasco, C., Robertson, K., & Schmitter-Edgecombe, M. (2015). Automated detection of activity transitions for prompting. IEEE Transactions on Human-Machine Systems, 45(5), 575585. PubMed ID: 27019791 doi:10.1109/THMS.2014.2362529

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gale, D., & Shapley, L.S. (1962). College admissions and the stability of marriage. The American Mathematical Monthly, 69(1), 915. doi:10.2307/2312726

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Granat, M.H. (2012). Event-based analysis of free-living behaviour. Physiological Measurement, 33(11), 17851800. PubMed ID: 23110873 doi:10.1088/0967-3334/33/11/1785

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Greenwell, R.N., & Seabold, D.E. (2014). The stable pairing problem. The Mathematics Teacher, 107(6), 446450. doi:10.5951/mathteacher.107.6.0446

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hibbing, P.R. (2019). PAutilities: Streamline physical activity research (Version 0.2.0). Retrieved from https://CRAN.R-project.org/package=PAutilities

    • Search Google Scholar
    • Export Citation
  • Hibbing, P.R., Ellingson, L.D., Dixon, P.M., & Welk, G.J. (2018). Adapted Sojourn models to estimate activity intensity in youth: A suite of tools. Medicine and Science in Sports and Exercise, 50(4), 846854. PubMed ID: 29135657 doi:10.1249/MSS.0000000000001486

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kawahara, Y., & Sugiyama, M. (2009). Change-point detection in time-series data by direct density-ratio estimation. In Proceedings of the 2009 SIAM International Conference on Data Mining (pp. 389400). Society for Industrial and Applied Mathematics: Philadelphia, PA.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Klein, T., & Giegerich, S. (2018). matchingMarkets: Analysis of stable matchings (Version 1.0.1). Retrieved from https://CRAN.R-project.org/package=matchingMarkets

    • Search Google Scholar
    • Export Citation
  • Kozina, S., Luštrek, M., & Gams, M. (2011). Dynamic signal segmentation for activity recognition. In Workshop Proceedings on Space, Time, and Ambient Intelligence, International Joint Conference on Artificial Intelligence (pp. 9398). Menlo Park, CA.

    • Search Google Scholar
    • Export Citation
  • LaMunion, S.R., Blythe, A.L., Hibbing, P.R., Kaplan, A.S., Clendenin, B.J., & Crouter, S.E. (2020). Use of consumer monitors for estimating energy expenditure in youth. Applied Physiology, Nutrition, and Metabolism, 45(2), 161168. doi:10.1139/apnm-2019-0129

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, S., Yamada, M., Collier, N., & Sugiyama, M. (2013). Change-point detection in time-series data by relative density-ratio estimation. Neural Networks, 43, 7283. PubMed ID: 23500502 doi:10.1016/j.neunet.2013.01.012

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lyden, K., Keadle, S.K., Staudenmayer, J., & Freedson, P.S. (2014). A method to estimate free-living active and sedentary behavior from an accelerometer. Medicine and Science in Sports and Exercise, 46(2), 386397. PubMed ID: 23860415 doi:10.1249/MSS.0b013e3182a42a2d

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Manlove, D.F. (2008). The Hospitals / Residents Problem. In M.Y. Kao (Ed.), Encyclopedia of Algorithms (pp. 390394). New York, NY: Springer.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matthews, C.E. (2005). Calibration of accelerometer output for adults. Medicine and Science in Sports and Exercise, 37(Suppl.), S512S522. doi:10.1249/01.mss.0000185659.11982.3d

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Needleman, S.B., & Wunsch, C.D. (1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3), 443453. PubMed ID: 5420325 doi:10.1016/0022-2836(70)90057-4

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Noor, M.H.M., Salcic, Z., & Wang, K.I.-K. (2017). Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer. Pervasive and Mobile Computing, 38, 4159. doi:10.1016/j.pmcj.2016.09.009

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nyan, M.N., Tay, F.E.H., Seah, K.H.W., & Sitoh, Y.Y. (2006). Classification of gait patterns in the time-frequency domain. Journal of Biomechanics, 39(14), 26472656. PubMed ID: 16212968 doi:10.1016/j.jbiomech.2005.08.014

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Preece, S.J., Goulermas, J.Y., Kenney, L.P.J., Howard, D., Meijer, K., & Crompton, R. (2009). Activity identification using body-mounted sensors—A review of classification techniques. Physiological Measurement, 30(4), R1R33. PubMed ID: 19342767 doi:10.1088/0967-3334/30/4/R01

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shahi, A., Woodford, B.J., & Lin, H. (2017). Dynamic real-time segmentation and recognition of activities using a multi-feature windowing approach. In U. Kang, E.-P. Lim, J.X. Yu, & Y.-S. Moon (Eds.), Trends and Applications in Knowledge Discovery and Data Mining (pp. 2638). Cham, Switzerland: Springer International Publishing. doi:10.1007/978-3-319-67274-8_3

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sprint, G., Cook, D.J., & Schmitter-Edgecombe, M. (2016). Unsupervised detection and analysis of changes in everyday physical activity data. Journal of Biomedical Informatics, 63, 5465. PubMed ID: 27471222 doi:10.1016/j.jbi.2016.07.020

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Troiano, R.P., McClain, J.J., Brychta, R.J., & Chen, K.Y. (2014). Evolution of accelerometer methods for physical activity research. British Journal of Sports Medicine, 48(13), 10191023. PubMed ID: 24782483 doi:10.1136/bjsports-2014-093546

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yamanishi, K., & Takeuchi, J. (2002). A unifying framework for detecting outliers and change points from non-stationary time series data. In Proceedings of the Eighth ACM SIGKDD, International Conference on Knowledge Discovery and Data Mining (pp. 676681). New York, NYACM.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 854 759 71
Full Text Views 19 8 1
PDF Downloads 12 6 1