Thrust Rationale, Organization, and Strategy: In Thrust 1, we are developing systems that augment the physical and sensory performance of a surgeon during the performance of a procedure. These systems are designed to work cooperatively with the surgeon in a dynamic or context-dependent fashion.

Work elsewhere on surgical assistance falls into two broad categories: physical assistance and visualization assistance. Such systems can provide position scaling, limited force feedback to the human operator, and high-quality visual displays. A variety of large-scale, enhance visualization surgical systems have been developed, the most prominent of which is now the da Vinci Surgical System marketed by Intuitive Surgical. This is a telemanipulated device that provides position scaling and stereo endoscope display, but currently no force feedback, artificially-generated visualization enhancements, or context-based operator assistance.

Our work differs in many respects from other efforts. First, we have focused much of our effort on direct manipulation assistance, whereby the surgeon and a single augmentation device “share” a surgical instrument. Our results suggest such a system can be as effective as, and perhaps more readily accepted by clinicians than, telemanipulation systems. Second, we are taking an integrated approach to visualization, sensing, and augmentation: most of the systems we envision will use intra-operative data to guide physical and sensory augmentation. Third, we include the broader context of modeling the entire procedure performed by the surgeon, allowing us to customize the physical and sensory augmentation provided at any time. Fourth, we are testing the effectiveness of such enhancements in both direct and telemanipulation systems from a human factors perspective.

Our strategy for Thrust 1 can be described under two major headings: (1) devices and systems; and (2) modeling and registration. For devices and systems, we focus on: (a) robot and display technology development; (b) sensor technology to image the local neighborhood of an interventional instrument with high accuracy and resolution, and in (near) real-time, but with low impact on the patient and/or surgeon; (c) systematic testing of human-machine performance using these technologies; and (d) system modularization. For the latter, we work with the infrastructure support of the ERC to refine and modularize our instruments to create ERC core infrastructure.

For modeling and registration, we focus on: (a) modeling and registration of the physical surgical field using available sensor information; (b) modeling and registration of the surgical procedure with the actions of the surgeon; and (c) developing models of human interactions with sensory and physical feedback systems. Here we have four principle challenges: (a) real-time, multi-model deformable registration, particularly from imaging devices such as the surgical microscope, endoscopes, ultrasound; (b) modeling surgical procedures in order to relate the state of an intervention to the appropriate assistance to be offered; (c) registration or recognition of surgical actions and events in order to register a model of a procedure to intra-operative performance; and (d) understanding how human dynamics, psychophysics, and perceptual models can be integrated into system design.

To meet these challenges, we have organized Thrust 1 into four tasks: testbed applications development, testing, and validation (Task 1); sensing, registration and visualization (Task 2); high-level surgical assistant development (Task 3); and human-machine performance evaluation (Task 4). Task 1 contains most of the application-specific “testbed” research and development. Task 2 deals with sensing, as well as modeling and registration and related problems in visualization. Task 3 addresses “high-level” modeling and integration issues. Generic system development issues are addressed in the context of supporting research and development of these three tasks. Task 4, which was added in Year 6, addresses the issue of developing quantitative methods for measuring or evaluating performance-enhancing systems, which cuts across all the other tasks and provides a human factors “infrastructure” for the entire center.

Relationship to the broad strategy and other thrusts: Thrust 1 makes two important contributions to our research strategy. First, it provides a strong motivation to think far beyond current technology and practice on all fronts. From a systems perspective, Thrust 1 is challenging because of its real-time integration of sensing, robotic assistance, and display. From an applications perspective, it is challenging because of the physical scale and delicacy of interventions. From an human factors perspective, it is challenging in that it proposes to provide context-sensitive, interactive assistance.

Thrust 1 also provides a strong systems and integration focus for the basic engineering vision of the Center. In order to achieve the goals of Thrust 1 across its multiple testbeds, we must be able to integrate a family of robotic devices seamlessly with a family of sensing and visualization technologies, and do so within the structure of different procedures on different organ systems.

As our technology matures, ideas developed in Thrust 2 are being incorporated into Thrust 1 and vice versa. This is exemplified by recent NIH grant submissions for high-precision eye interventions. This work combines many “Thrust 2” ideas, such as preoperative registration and targeting and pre-operative/intra-operative registration, with “Thrust 1” ideas such as online registration and human guidance.