My own interest in human motion began as an undergraduate studying to be a physical education teacher at Carnegie College in the suburbs of Leeds, England. As a gymnast with an interest in physics and mathematics, I was intrigued by the way twisting somersaults are created. In 1968, together with a few fellow gymnasts, I began some very rudimentary experiments with strings of flash light bulbs held over body segments by strips of elastic (Figure 1). These were powered by a small battery hooked to the subject’s waist. We recorded various skills performed on a trampoline and used an early one-inch Ampex tape recorder to examine the recordings in a qualitative fashion. These were early days, and our “experiments” were far from research. Biomechanics, as a subdiscipline of physical education, had yet to be introduced, and “motion capture” per se was decades ahead, but for me, it was the beginning of something much bigger.
—Incandescent body segment marker (1968–1969).
Citation: Journal of Applied Biomechanics 37, 6; 10.1123/jab.2021-0300
In 1968, I began a master’s degree in physical education at Michigan State University (MSU). My academic advisor was Dr William “Bill” Heusner, an Olympic swimmer (1948), exercise physiologist, statistician, and swimming coach. He recognized my interest in the mechanics of twisting skills, and he encouraged me to pursue it. He introduced me to high-speed cameras, and we put together a timing device for synchronizing several cameras.1 It was at MSU that I began to work on the mathematics required to reconstruct motion from synchronized high-speed images. I concluded my work at MSU by writing a thesis describing how to reconstruct human joint locations in 3 dimensions using image coordinate data recovered from synchronized, 16-mm cameras and elementary trigonometry,2 but this initial effort was less than rigorous. Bill wanted me to stay on at MSU and complete a doctorate under his tutelage, but the focus of his own work was exercise physiology, and I made it clear that I was more interested in measuring and modeling human motion. To his credit, Bill realized there were others who were better prepared to meet my needs. He exposed me to the work of Dr Thomas Kane at Stanford University (whose work with falling cats had just been published in Life magazine3,4), and he also advised me of the efforts of one of his own former students: Dr Richard “Dick” Nelson. He said that Dick was creating a new program devoted to biomechanics of sport at Penn State University (Penn State) and added that he thought this new program would be a good match for my specific interests and my previous academic preparation.
The Penn State Connection
I started my PhD program in the Penn State Biomechanics Laboratory in September 1970 together with Dave Dainty. We were the fourth and fifth PhD students to enter the program. Immediately, I began to take courses in basic mathematics, engineering, and computer science. Frankly, at that time, the new biomechanics program was still in its infancy, and we, the students, were spending time looking for appropriate classes to take; furthermore, as we reported back to Dick, he was learning from our experiences and forming a foundation for his nascent program. In this regard, I think Dick’s students did much of the “leg work” required to define the program. We found classes we wanted to take, and Dick gave us free rein to take them. This worked very well for me, and I have used a lot of the material I acquired during my time on the Penn State campus.
When we took our first engineering classes, our presence was received with some skepticism. As the initial roll was called, I can remember how we were questioned about our affiliation: “Phys. Ed.?” Other students in the class found this humorous, but we stood our ground, and after passing classes with flying colors, we became known as the “Super Jocks,” which we appreciated and accepted as a mark of respect. After a while, we became well known to the engineering faculty, and ties were established between the Biomechanics Laboratory and various engineering programs; however, our freedom was not without limits, and there were bumps in the road.
Although it was never discussed, Dick was obviously constrained by the other members of the physical education faculty. They wanted everyone in Dick’s program to have a “well-rounded background” in physical education; thus, every PhD candidate had to take a (dreaded) “Candidacy Examination.” As I recall, there were 10 “core” areas in which each PhD candidate was expected to demonstrate competency by passing this exam. Among the core areas were history of sport, the psychology of athletes, growth and development of school children, and so on. If a candidate failed to pass any one area, he/she was required to take a remedial class to make up for the “deficiency.” I am sure Dick had to toe the line in this regard, but from my perspective, I was wasting precious time. There were so many other classes I wanted to take—and in retrospect, these programs would have been far more useful to my subsequent career. I have no idea where things stand today, but I do hope that this constraint has been removed from the current program.
Apart from the external forces limiting our activities, we were not immune from issues of our own making. I recall feeling very good about my capabilities after taking a class in linear algebra (a class that provided information I found extremely useful when diving into the photogrammetry literature), so I decided to tackle a class on tensors—“multidimensional linear algebra.” It did not take me long to realize that I was in way over my head, so I dropped the class almost immediately. Such was our learning process!
I recall one other incident that caused some angst for Dick. By the second year, I had begun to develop some simple models of human motion, and I needed to process data recovered from 16-mm film using the FORTRAN programming language. At that time, any significant data processing had to be performed on an isolated IBM 360 mainframe located in the “Computation Center” on the far side of the campus. To use this facility, it was necessary to have an account associated with the student’s department. These accounts were issued to maintain a degree of control over the use of the facility by the various departments, and the amount of time allocated to each account reflected the anticipated use by each department. For obvious reasons, the engineering departments were given much larger accounts than the physical education department. When I asked Dick for an account number, he happily provided me with the number he had been given by the physical education department … an account used by the entire department for the sole purpose of maintaining grades! It took me just 4 days to deplete the account, and Dick was summoned by Dean Scannel to explain what was going on: “Jocks” were not expected to program computers! Such were the bumps in the road that Dick had to negotiate.
Some of us also had the opportunity to program a “state-of-the-art” Adage vector graphics computer located in the Computation Center. We learned how to draw 3-dimensional, geometric figures that represented body segments, and we learned how to apply cascaded transformations (rotations, translations, and scaling) to link these “rigid bodies” together so a movement of a thigh, for example, brought along the shank and foot (Figure 2). We also learned how to drive these graphic models with raw data and mathematical functions (these were the early days for computer graphics; rendered surfaces needed far more processing power than we had at the time). All of this tied in well with the classes we were taking in advanced dynamics and computer science, and it proved invaluable to me later in my career. Meanwhile, at night, we also learned how to play some of the first computer games on this very expensive toy! Once again, we made our presence known on the campus. Indeed, “Jocks” did program computers!
Output produced by an analytical model of the lower extremity using an Adage AGT-30 interactive graphics terminal (1973).
Citation: Journal of Applied Biomechanics 37, 6; 10.1123/jab.2021-0300
During my time working at the Penn State Computation Center, I got to know Professor Dan Bernitt, who was responsible for managing that facility. In late 1973, I told Dan, quite incidentally, that Penn State was about to host the 1974 Big 10 gymnastics championships and that this would be followed by the National Collegiate Athletic Association National Championship a few weeks later. I asked him if it might be possible to score these meets by computer, and this led to a very successful collaboration. During the regular (1974) season, we developed a networked system tied to the IBM mainframe in the Computation Center. Remote terminals were located on the floor at Rec. Hall at Penn State where scores were fed into the IBM machine. Using a set of APL functions developed by Dan, the scores could be saved and processed in real time. Additional terminals were provided for the press, and the participating coaches and gymnasts, so they could monitor the scoring as each routine was scored. This revolutionized the scoring process. Initially, the 1974 Penn State dual meets were scored in parallel with the regular team of scorers—who used pencils and paper. As the season progressed, everyone became confident that we could do the job, and at the end of the season, the 1974 Big 10 Championships and the National Collegiate Athletic Association National Championship were the first to be scored by computer. Everyone involved hung around the various terminals, and coach Gene Wettstone was ecstatic. Previously, he had to wait several minutes to find out which team had won and who had won the all-around competition and the various events; moreover, the manual scoring process could produce erroneous results that required embarrassing reversals, but now we could tell him exactly what score was needed for a particular gymnast (or team) to win an event. However, one group was very upset by our efforts: the judges. They soon learned that not only could we examine the progress of each event in real time but we could also detect systematic biases in the judging … in real time. Indeed, “Jocks” could program computers! Later, these events were described in the magazine “Gymnast”5 and at a professional conference in Italy.6 Bear in mind that it would be another 3 years before the Apple II became available (in 1977) and another 7 years before IBM introduced the IBM-PC (in 1981).
This said, the Biomechanics Laboratory did have its own small minicomputer, a model HP 2115A with an 8K memory. Input was provided by paper (or more robust Mylar) tapes, and output was fed to a Teletype printer that rattled out characters one at a time. This computer had a (literally) volatile core memory. If power was lost, the entire machine had to be rebooted. This involved toggling 32 instructions into the machine, one at a time, using 16 switches located on the front panel of the machine. This short program (the “bootstrap loader”) was sufficient to read a more extensive set of instructions (a very simple operating system) that was stored on Mylar tape. A two-pass, FORTRAN compiler was also available on Mylar tape, but this had to be loaded after everything else was up and running. It was quite a process, but it gave the students who learned how to use it a good understanding of how computers work. Again, I found this knowledge to be very useful as my career developed.
In regard to the minicomputer and other instrumentation in the biomechanics laboratory, it would be inappropriate not to recognize the significant contributions made by Mr. Ken Petak, our “resident genius,” who wrote low-level programs in machine code that interfaced with various devices used in the laboratory (today, we would refer to these programs as “drivers”). Ken was responsible for developing much of the custom instrumentation used by students and faculty alike. Tragically, Ken died in a traffic accident before I left Penn State. It was a great loss for Dick and everyone else who depended on Ken’s expertise. Over the years, I have learned that most facilities have a “Ken Petak” or even a “Ken Petak Department.” They exist in all types of research facilities, big and small, academic, corporate, and government. They are the unsung heroes of research, and I have great respect for their contribution to science. It did not take me long to learn that it was in my best interest to befriend the technical support staff wherever I worked. I have no doubt that it took Dick a long time to replace Ken.
Looking back on the work Dick had to accomplish to kick start the Biomechanics Laboratory, it clearly extended beyond finding suitable classes for his students. He needed to establish local, United States (US), and international contacts. He brought people to the Laboratory from around the world, and he reached out to facilities beyond the main Penn State campus. I recall that on one occasion he worked with us to arrange a visit to the Air Force Medical Research Laboratory in Dayton, Ohio. This visit was a result of various literature searches that produced documents describing the significant work in biomechanics (and allied sciences) that was being conducted by the Air Force at this facility. Dick organized a convoy to Dayton, and another connection was made. In 1973, he also arranged for Bob Gregor and me to take a 2-term course in gross anatomy at the Hershey Medical Center. This was a unique experience for Bob and me, but for me, it extended far beyond my academic preparation. It found me my wife, Dorcas, a nurse at the Medical Center. We have been together now for 47 years. Such was Dick’s impact!
I think it would be fair to say that, at the start, none of us really knew what we were doing, but we knew where we wanted to be. As German rocket pioneer Wernher von Braun is said to have observed, “Research is what I am doing when I don’t know what I am doing, if I know what I am doing, it’s not research.” This could well have been the mantra for the Biomechanics Laboratory in those early years, but it paid off. Certainly, it was through Dick’s efforts in the late 60s and his persistence in the early 70s that “Biomechanics of Sport” became an established discipline. Others representing similar disciplines (kinesiology, ergonomics, human factors, space research, etc) had made significant contributions to the field, albeit indirectly, but Dick “pulled it all together” and created a focused program. I am proud to say that I was one of the early participants in Dick’s program. He pulled, we pushed—sometimes in different directions—but it worked.
Beyond Penn State—Stanford and a Career in “Motion Capture”
The program at Penn State boosted me on to a unique career. Not in the sense that I performed in any special way; more that it gave me a foundation for what was to come. In 1973, Professor Tom Kane took part in the Fourth International Seminar on Biomechanics at Penn State, organized by Dick and Dr Chauncey “Dewey” Morehouse. This was my first encounter with Tom, and I told him of my interest in modeling and measuring human motion. I also told him I was aware of his work with falling cats and the motions of weightless astronauts. After listening to me describe my own interests, he told me that we were working on the same problem. At Stanford, he was working for NASA on twisting and turning in space, whereas I was trying to understand how someone on a trampoline could perform twisting somersaults. He went on to say that the only difference in our work was the time element—his subjects had all day to perform the required maneuvers; mine had only 2 seconds! He suggested that I work with him at Stanford, so, 1 year later, my new wife and I moved west to the Stanford campus.
Tom took me under his wing for the next 4 years (1974–1978), and like Dick, he gave me great freedom to pursue my interests. From 1974 to 1976, I served as his teaching assistant, and he gave me free rein to take classes that interested me. He also gave me free access to a computer that allowed me to complete work on my Penn State degree. At Stanford, I took core classes in dynamics, numerical methods, material properties, and fluid mechanics. The classes I took at Penn State prepared me well for this experience, and in 1976, I finished my MS in applied mechanics. I also published my first papers describing my work on Motion Capture7,8 and, in 1977, presented a paper in Copenhagen that described a model of human motion, displayed on an Adage graphics computer, that I had developed with Tom.9
After finishing my degree at Stanford, I turned my focus back to my Penn State degree and continued to work as a teaching assistant with Tom and his colleagues. In 1979, I defended my dissertation on the Penn State campus and then took a position with the General Motors Research Laboratories in Warren, Michigan.
My dissertation was published in 1981.10 It was well received, and since it was published, it has been purchased by more than 120 individuals, companies, and academic institutions in the US, Europe, Australia, and Asia. I know Dick was under a lot of pressure in the late 70s because he had let some of his students take several years to finish their doctoral degrees, but from my perspective, it was time well spent. Dick’s tolerance was greatly appreciated.
My dissertation described the mathematical foundation for what is known today as “Motion Capture.” The raw data used to confirm that the proposed tracking algorithms worked were collected from high speed images recorded at Penn State in 1973 using the test setup shown in Figure 3. Incidentally, the term “Motion Capture” was introduced in the mid-90s by animators working in Hollywood. We introduced them to the technology while I was working at Motion Analysis Corporation in the mid-80s. Later, in 1990, I worked with Hollywood animators to produce a data-driven, animated sequence (“The Golden Runner”) that was used by the US Olympic Committee to solicit the Olympic Games for Atlanta.11 Subsequently, my dissertation involved me in a multitude of disciplines that extended far beyond animation and biomechanics. In 1982, I worked with colleagues from the Massachusetts Institute of Technology on a baseball study that was published in “Science '82.”12,13 The original objective (specified by NBC-TV) was to measure the trajectory of pitched baseballs, in real-time, during the 1982 World Series, but this proved to be impractical at that time, so we focused on the trajectory of curveballs thrown by Scott McGregor of the Baltimore Orioles. After publication, this work was described on the NBC Today Show and in various national newspapers, Newsweek,14 and several books. (Note to future PhD candidates: baseball research has a lot more followers than car crash investigations; furthermore, it can be hard to defend even the most rigorous research if the results contradict folklore!)
—Experimental set-up for motion capture at Penn State (1973). Note the Walton timer1 on table in the foreground and the control points used for camera calibration.
Citation: Journal of Applied Biomechanics 37, 6; 10.1123/jab.2021-0300
In 1985, I received a call from Motion Analysis Corporation, a small, high-tech start-up located in Santa Rosa, California. They told me that they were using a video camera attached to a Sun workstation to measure ankle motions in 2 dimensions for Nike, but Nike wanted a system that could measure ankle motions in 3 dimensions. They told me that Nike had referred them to me because Nike was using my doctoral work to accomplish this task using high-speed cameras and 16-mm film. Shortly after this conversation, we moved back to California, and I began to adapt my software for use with video cameras.
I left Motion Analysis in September 1988; at the time of my departure, I was Vice-President of Applications Engineering. On October 1, 1988, I formed my own company, 4DVideo. The company recently celebrated its 33rd anniversary. Frankly, I’ve enjoyed the freedom, flexibility, and diversity that this move provided, but in the early days, it was touch and go. People told me I was walking a fine line between courage and stupidity, but with the help of family and friends, I managed to get the company up and running, and contracts began to arrive, first from the automotive industry, then from NASA and the military. This gave the company room to breathe and a degree of stability. Meanwhile, I organized and sponsored a minisymposium on image-based motion measurement, held during the First World Congress on Biomechanics.15 I also helped to organize several conferences for the Society of Photo-Optical Instrumentation Engineers (SPIE),16–21 and my activities began to receive interest from others working in electronic imaging.22 In 1994, I was invited to participate in the 21st International Congress on High Speed Photography and Photonics held in Taejon, Korea.23 Since then, I have participated in all subsequent meetings of this group, and from 2006 to 2016, I served as the US National Delegate to this congress. In 2001, I was made a fellow of SPIE, and in 2010, I was recognized with an Outstanding Career Achievement award by the Kinesiology Department at MSU.
At 4DVideo, I have applied the techniques I developed at Penn State to a very wide spectrum of problems. By the late 80s, I was well aware that the methods I had developed could be used to examine far more than human motion, so I began to lecture on the use of cameras as 3-dimensional transducers: devices that can convert a signal from one form to another,24,25 much as an accelerometer or strain gauge converts an electronic signal from one form to another. I described how a photo or video stream can be converted to displacements, velocities, frequencies, and other data forms. By way of illustration, I have used motion capture technology to quantify human gait, the recoil of military firearms, car crash studies, the development of musical instruments (keyboards), tidal flow in the Mississippi river, the design of toys, brake vibration, vibration issues in hard disk drives, the performance of racehorses, predator–prey relationships between insects, vibration and handling characteristics of military tracked vehicles and earth moving equipment, the kinematics of ocean buoys, the deployment of oil rigs in the Pacific Ocean, sperm fertility measurements, wave tank experiments, ergonomics on auto assembly lines, wind tunnel testing and space suit design for NASA, patent litigation, bank robberies, injury biomechanics, engine mount design, the schooling behavior of fish (in the Monterey Bay Aquarium), expert witness testimony, and the deployment of Sidewinder missiles from the wings of F-18 jets. Examples are shown in Figures 4 and 5.
—Motion capture to determine the range of motion of space suits (1988). These tests were conducted underwater to simulate weightlessness, and (black) racquetballs were used to mark reference points on the space suits.
Citation: Journal of Applied Biomechanics 37, 6; 10.1123/jab.2021-0300
—Motion capture setup to measure the 3-dimensional kinematics of road wheels and steering linkage due to brake vibration (1997). The tests used 7 synchronized cameras to correlate steering wheel vibrations with the steering linkage and road wheel measurements. The tests were conducted on a 2-mile test track at speeds in excess of 100 miles per hour. Output also included dynamic camber, caster, and toe measurements.
Citation: Journal of Applied Biomechanics 37, 6; 10.1123/jab.2021-0300
In 1972, I photographed a cluster of 18 golf balls hung from a 4′ × 8′ sheet of plywood in Dick’s new biomechanics laboratory. The question: “Could I create a 3-dimensional scale that would allow me to locate these golf balls in 3 dimensions?” It was from these early experiments that all else flowed. I still have the output sheets from the IBM 360 that showed it could be done. I have enjoyed a remarkable career, and I am incredibly grateful to those who made it possible. Much of my success can be traced back to the 4 years I spent on the Penn State campus. Dick Nelson … I thank you for providing me with that opportunity.
And by the way, “Jocks” do program computers!
References
- 2.↑
Walton JS. Photographic and Computational Techniques for Three-Dimensional Location of Trampolinists. [Master’s thesis]. Michigan State University; 1970.
- 4.↑
Kane TR, Scher MP. A dynamical explanation of the falling cat phenomenon. Int J Solid Struct. 1969;5:663–670.
- 6.↑
Bernitt DL, Walton JS, Bader FS. Large scale gymnastics championships: an on-line, interactive collection and analysis of scores. Proceedings of the seventh international conference on APL; 1975. Pisa, Italy.
- 7↑
Walton JS. Cinematography as a tool in biomechanics of sport. Proceedings of the 117th Technical Meeting of the Society of Motion Picture and Television Engineers (SMPTE); 1975. Los Angeles, California.
- 8.↑
Walton, JS. Close-range cine-photogrammetry: Another approach to motion analysis. Proceedings of the International Conference on Sport Science: International Clinic on High-Speed Biomechanics Cinematography; 1978. Edmonton, Alberta.
- 9.↑
Walton JS, Kane TR. Interactive computer graphics—a new coaching aid. Proceedings of the 6th International Seminar on Biomechanics: “Biomechanics VI”. 1977;439–444, Copenhagen, Denmark.
- 10.↑
Walton JS. Close-Range Cine-Photogrammetry: A Generalized Technique for Quantifying Gross Human Motion. [Doctoral dissertation]. Pennsylvania State University; 1981.
- 11.↑
Vasilopoulos A. Atlanta gets the gold: Multimedia presentation helps Atlanta win bid as host city to 1996 summer Olympic Games. Comput Graph World. 1990;13(12):83–84, 86, 89, 91.
- 13.↑
Walton JS, Watts R. Gravity’s rainbows: The kinematics of baseballs and bodies. Proceedings of the 149th Annual Meeting of the American Association for the Advancement of Science (AAAS); 1983. Detroit, Michigan.
- 15.↑
Walton JS, ed. Proceedings of First World Congress of Biomechanics: Mini-Symposium on Image-Based Motion Measurement; 1990. SPIE Volume 1356, La Jolla, California.
- 16.↑
Davidhazy A, Etoh T, Johnson CB, Snyder DR, Walton JS, eds. Proceedings of 42nd Annual Meeting of the Society of Photo-Optical Instrumentation Engineers (SPIE): Conference on Ultra- and High-Speed Photography and Image-Based Motion Measurement; 1997. SPIE Volume 3173, San Diego, California.
- 17.
Walton JS, ed. Proceedings of Electronic Imaging '99: Conference on High-Speed Imaging and Sequence Analysis; 1999. SPIE Volume 3642, San Jose, California.
- 18.
Frank AM, Walton JS, eds. Proceedings of Electronic Imaging 2000: Conference on High-Speed Imaging and Sequence Analysis II; 2000. SPIE Volume 3968, San Jose, California.
- 19.
Frank AM, Walton JS, eds. Proceedings of Electronic Imaging 2001: Conference on High-Speed Imaging and Sequence Analysis III; 2001. SPIE Volume 4308, San Jose, California.
- 20.
El-Hakim SF, Gruen A, Walton JS. eds. Proceedings of Electronic Imaging 2003: Videometrics VII; 2003. SPIE Volume 5013, Santa Clara, California.
- 21.↑
Beraldin JA, El-Hakim SF, Gruen A, Walton JS, eds. Proceedings of Electronic Imaging 2005: Videometrics VIII; 2005. SPIE Volume 5665, Santa Clara, California.
- 22.↑
Johnson H. The mechanical engineer and the transition to image analysis. Adv Imaging. 1990;5(11):52–56.
- 23.↑
Walton JS. Image-based motion measurement: New technology, new applications. Proceedings of the 21st International Congress on High-Speed Photography and Photonics (ICHSPP '94); 1994. SPIE Volume 2513: 962–880, Taejon, Korea.
- 25.↑
Walton JS. Cameras as displacement transducers—another approach to motion measurement in three-dimensions. Test Eng Manag.2008;70(2): 10–13.
