Five years ago, the Whiting School of Engineering and the Applied Physics Laboratory decided to formally combine their research and design expertise for a wide range of cutting-edge projects. The results have been nothing short of groundbreaking.
In a small workroom at Hopkins’ Applied Physics Laboratory (APL), Jacob Vogelstein, PhD ’07, picks up a red and silver mechanical hand—a beautiful collection of joints, pulleys, wires, and alloys. “In a human hand, many of the muscles and tendons that control the hand itself are actually located in the forearm,” explains the biomedical engineer.
But for someone who has lost only a hand, the intricate mechanisms, motors, and gears needed to artificially operate the mechanical fingers, thumb, and wrist must be contained within the hand itself— a feat of engineering that even evolution hasn’t managed to pull off.
“DARPA-Hard”
With this sleek prosthetic device, known as the Proto 2 Intrinsic Hand, Hopkins engineers are striving to improve upon nature itself. The Proto 2 houses a dizzying number of components, all of them intricate, powerful, and miniature. “Some of the fine motor assembly is done in Switzerland, by craftspeople who also manufacture and assemble gears for watchmaking,” Vogelstein says. “Our tiny motors have the same level of complexity and need the same level of precision.”
Complexity is commonplace at APL’s Revolutionizing Prosthetics 2009 (RP 2009) facilities, as dozens of researchers are working to create and fine-tune an artificial hand (and hand/arm combinations of varying lengths) that will offer previously unimagined function and use for potentially thousands of amputees. One of the main intellectual and research strengths that Laurel, Maryland–based APL was able to bring to the project was a newly rejuvenated collaborative relationship with Hopkins’ Whiting School of Engineering (WSE).
“I was very excited when I heard that the project had been awarded,” says Whiting School Dean Nick Jones. “It struck me as exactly the type of work we had always wanted. It’s a really, really challenging project, and the only way of pulling it off was by bringing together people with different expertise and talents.”
Stuart Harshbarger, team leader for the RP 2009 program, says that one crucial part of the project’s success has been the collaboration between APL’s researchers and staff and the faculty at the Whiting School. “This program certainly wouldn’t have been a success without the collaboration with Homewood, as well as the other partner groups and schools,” says Harshbarger, who also serves as program manager and systems integrator for the project.
RP 2009 is an ambitious multiyear endeavor, sponsored by the U.S. government’s Defense Advanced Research Projects Agency, or DARPA. Led by APL (which oversees a group of some 30 partner organizations, totaling more than 100 engineers) since 2006, RP 2009 aims to redefine the state of the art for upper-limb prosthetics. After the initial phase of the program demonstrated immense success with Proto 1, a first-generation artificial limb and hand with neural control, and showed the feasibility of the more advanced Proto 2 systems, DARPA approved the subsequent $38 million expenditure to develop the final limb.
Engineering projects come in varying degrees of difficulty. One of the top echelons of challenge comes in contracts generated by DARPA, which brings the vast financial resources of the nation to bear on problems that may seem insurmountable, and generally benefit national defense interests. These projects—almost always secret, very expensive, and always very complex—have their own difficulty designation among engineers: DARPA-hard. This rating is the highest degree of difficulty, and consequently is an alluring challenge to the world’s best and brightest researchers, including those at both APL and the Whiting School.
The current state of collaboration between APL and WSE represents a major advance of its own. As recently as a decade ago, the school and the lab would only occasionally interact, owing in part to the differing philosophies, cultures, and missions of each institution.
“In the past there was too much of a tendency for people on both sides to say, ‘Oh, we can’t do that,’” says John Sommerer, director of science and technology and chief technology officer at APL.
“There was a perception that Homewood was not interested in applied research—that they were only interested in discovery. There was some truth in that,” says Allan Bjerkaas, associate dean of the Whiting School’s Engineering for Professionals (EP) program since 2001 and before that a full-time scientist at APL for 31 years. “Now the faculty at the Whiting School are more open to applied research, and that helps.”
“We’re leveraging the strengths of the two organizations, and it’s alowing us to pursue all sorts of work on larger problems, larger than either APL or Homewood could pursue individually.” –Stuart Harshbarger, APL team leader for Revolutionizing Prosthetics 2009
The formal mechanism for change came in 2003 with the establishment of the WSE/APL Partnership Program, which grew out of a task force organized by university President William R. Brody. The program laid out steps and policies aimed at encouraging the two institutions to draw on each other’s strengths.
“Historically, most of the collaboration between APL and the Whiting School has been driven by principal investigators (PI),” says APL’s Harshbarger, who also teaches in the EP program. “We are seeing the transition from that PI-centric collaboration to a much larger scope. We’re leveraging the strengths of the two organizations, and it’s allowing us to pursue all sorts of work on larger problems, larger than either APL or Homewood could pursue individually.”
In addition to the RP 2009 program, WSE and APL have had success collaborating in other fields, including two of vastly different scope. First is APL’s civilian space division (which operates separately from the large and classified work done by the military space division); Whiting School faculty and graduates regularly participate in the design and deployment of a variety of satellites and space science instrumentation and missions. These have included, most recently, the Messenger mission to Mercury, and—at the far reaches of the solar system—the New Horizons mission to Pluto and the Kuiper Belt.
Back on Earth, APL and WSE researchers are working to invent revolutionary human speech recognition and analysis tools at the Human Language Technology Center of Excellence. Funded by the U.S. Department of Defense, this challenging project involves designing software and hardware systems that will be able to listen to a human voice— in any of several dozen languages—and provide a coherent, complete report of what that voice is saying.
“We’re an R&D institution,” says APL’s Sommerer. “From our perspective, the academic divisions at Hopkins are closer to the bleeding edge. And importantly, they come with networks to other academic enterprises. At a place like APL, even with the advent of technological globalization, it’s difficult for us to engage with the international community. The Whiting School represents an opportunity for us to have intimate contact with an academic institution.”
Building a Better Hand
When the silver-gray finger joints of the Proto 2 hand clench and unclench, the similarity to a human hand is uncanny. This version is an exponential evolution from the crude hooks and lifeless artificial hands commonly used now as prosthetics for people who have lost their hands and arms.
But there’s still plenty of work left to be done, because DARPA’s challenge is all-encompassing: First, build a modular artificial hand and arm system that can be fitted to any amputee, whether they are missing only a hand, a hand and forearm, the arm below the elbow, the arm above the elbow, or the entire arm. As for the arm and hand: They should be self-contained, have an almost-human range of motion, and not weigh more than a normal arm, even though they will contain a vast array of motors, batteries, and gears—all notoriously heavy pieces of equipment. The prosthetic hand should be able to perform almost exactly like a real human hand, and the hand needs to provide feedback to the user about whatever it is that is being touched, held, or manipulated. Is it hot? Is it soft? The user needs to know.
And the user needs to control the prosthesis through neural signals, in real time. When the user thinks, Reach out and pick up that peach, the arm and hand should do it at that exact moment, just like a real arm. That’s one step in solving one of the most serious challenges of RP 2009: translating the motor commands from the prosthetic’s wearer into real-time movement, and sensory data from the arm into feedback for the user. The difficulty is in collecting these huge amounts of data, analyzing it, having the software correctly interpret the signals, and then turning that decision into arm and hand motion with no noticeable delay—and that’s before meeting the requirement to provide near-instant tactile information back to the wearer.
To push the project forward, team leaders are tapping into the best research and minds, including many Whiting School faculty and graduate students. One of them is Nitish Thakor, a professor in Biomedical Engineering, who oversees several projects including sensors for touch, force, and temperature; skin-like cosmesis; prosthetic EEG control; and decoding neurons to control individual fingers and dexterous motions. Another is Ralph Etienne-Cummings, a professor in Electrical and Computer Engineering. He consults on the specialized neural circuits that produce rhythmic outputs to control motor systems. Allison Okamura, an associate professor in Mechanical Engineering, brings expertise in sensory feedback, a research field (known as haptics) that has recently made major advances.
In the lab at APL, Vogelstein is lead engineer for neural interfaces on the project. While the work is physically and mentally demanding (most researchers are working well into the night, nearly every night), Vogelstein says it’s also been, well, fun.
By way of example, he points to a Nintendo Wii video game system that sits on one of the worktables in the lab, and the two Guitar Hero video guitar-shaped controllers that rest on a chair. “We were looking for a way to test the neural controls for dexterity,” Vogelstein explains. “The historical training paradigms are very slow and basic. One of the things we need to do as we work is develop the next generation of training paradigms.”
The researchers and scientists needed something to test the fingers and control systems that would be used for the Proto 2 hand. Someone suggested Guitar Hero—a game based on the timed pressing of five color- coded buttons (located on the faux guitars) to the music of popular rock songs. The scientists made some customizations to the equipment, and now they have a perfect testing device.
“What we do is record signals from the residual muscles of an amputee, for example, the forearm muscles, if they’ve lost only a hand,” explains Vogelstein. “We decode the signals using our real-time software algorithms. The computer then uses the output from those algorithms to guide the Guitar Hero controllers directly, without any mechanical intervention.”
It’s an ideal solution; the tests are always the same, the game measures accuracy and response at just the necessary levels, and it’s an engaging process that keeps everyone interested (and sure beats a test tone). And it’s just one of the ways that researchers are designing a system that takes thought impulses and turns them into a lifting prosthetic arm, or a grasping prosthetic hand.
“We need to compete in the space of ideas and the collaboration with Homewood helps us do that. It also helps in the competition for human capital; we can hire smart people. We’d like to hire as many Whiting School graduates as we can.” –John Sommerer, APL’s chief technology officer
That requires a hybrid system, explains APL’s Harshbarger, one using sensors on a variety of nerves to capture as much information as possible. Sensors will take readings from noninvasive surface electrodes, from small implantable devices in residual muscle nerves, and from peripheral nerve and cortical areas—all done in varying combinations, depending on the motion—that are all fed into an analyzer. Algorithms developed by various universities and labs are applied, the outputs are combined, and the system makes the best available decision on what the prosthetic’s wearer wants to do—whether it’s pick up a stapler, wave to a colleague, turn on a coffee maker … or nail the guitar solo for “Sweet Child O’ Mine.”
A Win-Win
The new spirit of collaboration between WSE and APL is already having an impact on APL’s workforce, much to the delight of its leaders. Jacob Vogelstein is just one of five recent Whiting School graduates hired by the Lab.
“We need to compete in the space of ideas and the collaboration with Homewood helps us do that,” says Sommerer. “It also helps in the competition for human capital; we can hire smart people. We’d like to hire as many Whiting School graduates as we can. If we have a competitive advantage—if they did an internship here, for example—it gives us a really good chance to hire them.” Last summer, some 22 Whiting School undergraduates got a taste of what it’s like to work at APL, through a new summer internship program established by Johns Hopkins University Provost Kristina Johnson (see “APL’s ‘Advanced’ Placement”).
The Engineering for Professionals program (formerly EPP), which offers engineering course work and advanced degrees for working engineers, has been a perfect vehicle for building ties between both institutions. “APL senior staff have the opportunity to teach, and junior staff can pursue degrees part time in the Whiting School,” explains Bjerkaas. “With this collaboration, many people at APL know about the Whiting School who wouldn’t. In terms of education, it’s a very healthy, vibrant collaboration. Serious workforce development is a worthwhile contribution.”
There’s further cross-pollination through a joint appointment program for APL staff, which allows selected researchers to spend one day a week as a research professor at Homewood.
Vogelstein has a joint appointment in the Department of Electrical and Computer Engineering, and is a big fan of the intellectual breadth of being able to move back and forth between his workplace and alma mater: “The collaborations with the Whiting School are an excellent way to see what’s coming up, to see the next generation of technology while it’s still in the lab.”
“One of the things we’ve been talking about with APL,” says Marc Donohue, vice dean for research at the Whiting School, “is getting a complementary joint appointment in place, where Whiting School faculty would spend one day a week at APL. It would be a good way to promote even more interactions.”
There’s also a new mechanism for encouraging the exchange of ideas. The WSE/APL Partnership Fund is a $500,000 pool (administered by the Whiting School’s Donohue and APL’s Sommerer) that provides “seed money” to teams made up of researchers from both institutions. The money supports the work needed to put together a proposal that will lead to future outside funding.
“This goes to the model of what we believe collaborative research really is,” says Donohue. “We want two people to work together, like intermeshing fingers. Not feeding just information to one another but also ideas. It’s not something that’s revolutionary in concept,” says Donohue, “but it is in practice.”
Revolutionizing Collaboration
The practice of revolutionizing collaboration has proven to be a critical part of the Revolutionizing Prosthetics program.
“There are an immense number of materials and components to integrate,” says Harshbarger, reeling them off. “There’s the virtual environment to train the wearer of the prosthetic. There’s a modular mechanical limb system. There’s a modular neural control system. There’s the body attachment, which is very different below the elbow when compared to below the shoulder. It needs to be comfortable, and it can’t cause irritation.”
Harshbarger believes that it is the magnitude of these challenges that has inspired researchers to collaborate at every level. Another benefit of working on a piece of engineering with such an obvious quality-of-life outcome is the enthusiasm and interest it generates among scientists at both APL and Homewood.
“Everybody’s excited about this project,” says Vogelstein. “In grad school, each student is focused on one small piece of a puzzle. Revolutionizing Prosthetics is attractive because it offers people a chance to see how all the pieces fit together in a real-world application of science that directly benefits humanity and society. It makes it very easy to collaborate. Even people not officially on the program are happy to evaluate and give feedback because of the nature of the work. Almost every part of what we’re doing is going to be the first of its kind. No one has ever done this before,” Vogelstein says. “When we succeed, we will have advanced the state of the art by an order of magnitude over what it was before. It’s great to come to the lab knowing that.”