Dimitris N. Metaxas - US grants

9309917 Metaxes This is the first year of a three year continuing grant. This research addresses the problem of shape and nonrigid motion estimation using physics-based techniques and a general active vision approach. Almost all prior model-based shape and norigid motion estimation techniques assume prior data segmentation, simple parameterized models, fixed model size grids and a single viewpoint under orthographic projection. Real world images require far more sophisticated algorithms and models than the currently existing ones in order to robustly recover the shape and motions of the underlying objects. This research project will first explore new physics-based techniques for the fitting of models to image data in case of general orthographic, perspective, and stereo projections. Such data will include projections of complex objects with surface discontinuities appearing on the image as internal edges. Then, adaptive finite element techniques will be developed to improve the efficiency and accuracy of shape estimation through automatic local adjustment of the model's initial grid size. These efforts will be facilitated by the experience gained from the PI's previously developed physics-based modeling system for shape and nonrigid motion estimation.

The methods of rapid prototyping are ideally suited to rehabilitation devices. Because each person requires unique performance and function in a rehabilitation device, devices specific to each person must be rapidly designed and produced. This project is investigating a completely integrated approach to the design and prototyping of passive mechanical rehabilitation devices. The approach involves: the quantitative assessment of the form and performance of human limbs; the design of the assistive device; evaluation of the device using virtual prototyping; feedback from the consumer and therapist; actual prototyping of the device; evaluation of the function and performance of the device; and redesign based on performance. The contributions of the product include: the development of new computer-based tools for the assessment of human performance; a manufacturing technique for a new class of hyperelastic materials; the integration of tools into a rapid prototyping system for rehabilitation devices; and development of mechanisms for systematic evaluation of the final product.

The objective of this research is the development of methods for the automatic recognition of American Sign Language (ASL) utterances using as input the 3D shape and motion parameters of a subject's face, hands and arm. These parameters are extracted based on the use of computer vision techniques on relevant image sequences. The novel aspects of this research are: 1) use of 3D information from the vision-based analysis of the data that consists of the three-dimensional hand, arm and face shape and motion, 2) use of Hidden Markov Models (HMMs) to recognize the ASL structure at multiple levels, and 3) coupling of computer vision and HMMs to go beyond the limitations of both computer vision and HMMs. Novel computer vision methods will be developed based on the use of deformable models, visual cues, and knowledge of anthropometry to allow the accurate tracking of a human's face and upper limbs. The 3D output from the vision system will be used as input to the HMM for ASL recognition. To improve the robustness of the system, research on feedback mechanisms between the computer vision system and the HMMs will also be conducted. The final goal of this research is to demonstrate the feasibility of building an automated robust system with high recognition accuracy (recovery of sign sequences at the sign level) that is capable of handling the inflectional and derivational properties of ASL in a systematic way.

DESCRIPTION (Adapted from the applicant's abstract): Heart disease, a major cause of morbidity and mortality in the Western World, generally leads to abnormalities of heart wall motion. However, there are still major difficulties in clinical assessment of heart wall disease due to the use of conventional imaging techniques (e.g., CT, conventional MRI), the lack of sufficient resolution in the extracted data, the absence of computational techniques for automatic extraction of the three-dimensional heart wall motion parameters in a way that is "useful" to physicians, and the absence of a database of normal patients against which normal and abnormal hearts can be compared. The aim of this proposal is to use the recently developed at the University of Pennsylvania, magnetic resonance imaging (MRI) technique based on magnetic tagging ("SPAMM") for modeling and clinically analyzing the cardiac motion. In particular this proposal aims to: 1) develop methods for ventricular analysis and modeling, 2) apply the methods to a combination of previously acquired normal subject data and some additional data in order to construct a novel canonical heart motion model, including dependence on age, gender, race and body size, 3) incorporate the resulting normal database representation in the analysis and modeling tools program to create for an individual subject a 3D visualization of their heart displaying those areas which move abnormally, and 4) use the tool developed in #3 and apply it to a limited series of patients as a preliminary assessment of the utility of these methods and as a guide on how to improve the analysis and display methods. The hypothesis is that due to this type of data (MRI-SPAMM) which shows a much greater degree of precision in 3D, we will be able to reliably distinguish in a quantitative way the above types of disease that are expected to affect cardiac function in characteristic ways. We will test this hypothesis by an already available small representative series of patients, each with a well-defined heart condition. The results of the analysis will be tested against all other available evaluations of the patients studied.

The University of Pennsylvania and Boston University are collaborating on the establishment and maintenance of resources for research in sign language and gesture. The goal of this project is to make available several different types of experimental resources and analyzed data to facilitate linguistic and computational research on signed languages and the gestural components of spoken languages. Activities in the project include the following:

* A facility for collection of video-based language data will be established, equipped with synchronized digital cameras to capture multiple views of the subject.

* A substantial corpus of American Sign Language video data will be collected from native signers and made available in both compressed and uncompressed forms.

* Significant portions of the collected data will be linguistically annotated. The database of linguistic annotations will be made publicly available, along with the applications needed to access the database.

* Video data will be analyzed using computer-based algorithms, with analysis and software made publicly available.

Thus the project makes available sophisticated facilities for data collection, a standard protocol for such collection, and large amounts of language data. The combination of linguistic and computational expertise in this project will ensure scientific integrity of data collection, and will result in useful data for researchers in a variety of fields.

We are requesting $100,000 of NSF ILI support for LIVE: The Laboratory for Visual Environments at the University of Pennsylvania. With the requested NSF ILI support and School of Engineering and Applied Science matching funds, we would purchase 24 OpenGL Windows NT systems and networking infrastructure to support undergraduate teaching and projects in computer graphics, visual and virtual environments, and high speed multi-computer networking. Direct manipulation and programming of these concepts is the ideal setting for teaching and learning about complex phenomena. LIVE will upgrade an aging SGI facility, serve Computer and Information Science curriculum needs, and enhance the computer science aspects of a new undergraduate program in Digital Media Technology spanning the Engineering, Fine Arts, and Communications Schools at the University of Pennsylvania.

Abstract

IIS-9820794
Metaxas, Dimitris
University of Pennsylvania
$150,658 - 12 mos.

Coupling Deformable Model and Pixel Affinity Methods for Medical Image Segmentation and 3D Organ Reconstruction

This is the first year funding of a three year continuing award. This research develops new and efficient methods for segmenting radiological data of internal organs and reconstructing their 3D shape, to improve clinical practice and medical
education. These new methods allow the inclusion of image features such as patterns, textures and contours, to the storage and retrieval of data from the patient databases of the future. The methods we develop do not depend on the imaging modality used and can
be used for any type of internal organ. However, given the focus of our current research, we segment primarily the lungs and the heart and secondarily the brain. Such data are routinely used at the University of Pennsylvania. In particular we develop 1) methods for detecting and segmenting the boundaries of internal organs, 2) methods for detecting
abnormalities in the form of lesions, such as cancers and infections, 3) appropriate user interfaces so that the segmentation results can be superimposed on the radiological image, 4) 3D reconstructions of the shape of the internal organs using deformable models and splines, 5) the necessary evaluation procedures to test the accuracy of the segmentation methods and the efficacy of the user interface.

Procedural synthesis of natural and contextually appropriate gestures in embodied virtual human agents is challenging. Laban Movement Analysis (LMA) offers a descriptive system for human gesture qualities that fills the gap between pre-defined gesture playback systems and human animator intuition. A computational analog of LMA called EMOTE has been constructed whose parameters modify the performance qualities of arm gesture movements. EMOTE will be developed in several new ways:

* Connect EMOTE with an agent model so that an agent's affect, personality, and communicative needs set appropriate EMOTE parameters for gesture performance.

* Investigate motion analysis techniques for extracting EMOTE parameters from live dual or single camera views.

* Experimentally validate the automated acquisition of EMOTE parameters by using professional LMA notators for ground truth.

* Use the extracted parameters to create instances of parameterized actions which may be subsequently used for action, affect, and manner descriptions and, ultimately, for content-directed analysis of existing film or video material.

This study will help set synthetic agent animation techniques on a sound empirical footing, provide evidence that computers can in fact observe important motion qualities, and lead to strong connections between internal agent state and external behavior qualities.

DESCRIPTION (provided by applicant): Traditional cardiac imaging methods have only allowed us to focus on gross features of LV size and shape, due to lack of resolution (e.g., MR and ultrasound). However, heart disease affects the walls and the wall structure is complex (e.g., trabeculae and papillary muscles). These structures are usually ignored, due to lack of resolution, in cardiac shape analysis and cardiac functional assessment. Thus, no one has yet studied the functional and geometric changes between normal and diseased patients at a finer scale than simple gross measures, such as overall ventricular size and wall thickness. Recent developments in multidetector CT technology allow the acquisition of high resolution synchronized cardiac CT data within a breathhold. These new data allows us to assess the structure of the ventricles at an unprecedented level of detail. Given the above new developments, we propose to develop novel computational methods to characterize and quantify the geometry of the LV at levels which were not previously possible. These tools will be based on already collected data at NYU by Dr. Axel's (co-investigator) group. In particular, the specific aims of this proposal are as follows: 1) Develop tools for segmentation and visualization in 3D of the LV structure, including papillary muscles, trabeculae and valve attachments, at different phases of the cardiac cycle, 2) Develop tools for quantification of the architecture of the above structures and statistical analysis on their variations, 3) Perform an initial pilot assessment on normal subjects and some patients with high blood pressure. We will look at the effects of high blood pressure on both the overall structure and the detailed structure;Dr. Axel will provide the groundtruth and Dr. Madigan will provide the statistical analysis, and 4) Development of a database from our analysis results which will be made publicly available, and will be based on the ITK software toolkit (www.itk.org), to which the PI has already contributed many segmentation algorithms.

? DESCRIPTION (provided by applicant): Cardiac dyssynchrony deteriorates cardiac function and often cannot be treated effectively. The goal of this proposal is to develop and provide a new analysis technique for understanding the complex cardiac motion patterns (in time and space) of patients with cardiac dyssynchrony, with the hope of improving its treatment outcomes, specifically with respect to cardiac resynchronization therapy (CRT). CRT, the most effective treatment for dyssynchrony with worsening heart failure, significantly improves outcome in only ~66% of heart failure patients selected for the treatment, which is based on ECG criteria. Selection criteria based on imaging are essential to improving the success rate. Unfortunately, response rates are not higher with selection criteria based on echocardiography, the most popular cardiac imaging technique. Cine cardiovascular magnetic resonance (CMR) has the potential to better characterize dyssynchrony, as it shows cardiac mechanics and intramural wall motion with much higher spatial resolution than echocardiography. However, quantitative assessments of CMR have been mostly limited to global volumetric measures, which ignore most of the motion information captured by the images. For example, studies of a number of distinctive motion features of dyssynchrony (such as septal flash and apical rocking) have been confined to qualitative assessment, limiting inference of their potential utility for improving CRT treatment. To accurately quantify cardiac function through CMR, we have developed biomechanical models for describing cardiac function and machine learning technology for identifying morphological and functional patterns atypical for healthy hearts. We propose to combine these two technologies to accurately quantify cardiac dyssynchrony within the Left Ventricle (LV). Specifically, our methods will extract a rich description of LV motion and strain from the CMRs of a set of retrospectively selected subjects with synchronous or dyssynchronous LV motion. We will then use machine learning methods to identify local and global motion patterns specific to dyssynchrony. Finally, we will correlate these patterns to already existing clinical scores to find potentially predictive markers with respect to CRT outcome. We hypothesize that these markers will have a higher correlation to CRT outcome than current clinical markers alone. Identifying these markers will have the potential to further stratify the disease with respect to the expected outcome of CRT, which then can be used to derive new selection criteria that lead to higher success rates. The project will also disseminate our novel, data-driven methodology for quantifying that motion. Other research groups can apply our tools to specifically study dyssynchrony, as well as other cardiac diseases impacting LV motion in general.

Summary/Abstract In the presence of diseases such as ischemic heart disease (IHD), cardiac dyssynchrony deteriorates cardiac function and often cannot be treated effectively. However, while imaging methods such as cardiovascular magnetic resonance (CMR) can provide high quality images of the moving heart, conventional clinical quantitative analysis of cardiac function is largely limited to global function analysis of the left ventricle (LV), with only qualitative and subjective characterization of regional function. An obstacle to better quantification of regional function is the complex 3D structure and motion of the heart wall, which has typically necessitated time-consuming user-guided processing of the images to carry out the associated 3D-motion analysis. Recent advances in machine-learning (ML) approaches for image analysis are promising as new means to speed up the processing of cardiac images, as well as to analyze the underlying regional motion patterns. However, current Deep ML (DML) approaches to image analysis largely function as ?black boxes?, without clear indications of which features contribute most to the analysis results, thus limiting their clinical utility. In the initial funded period of this research project, we have been developing integrated approaches to the segmentation, 3D reconstruction, and analysis of CMR data, with application to the evaluation of cardiac dyssynchrony. Today, treatment of dyssynchrony in HF with cardiac resynchronization therapy (CRT) leads to improvement in only ~2/3 patients selected with conventional criteria (usually by electrocardiogram [ECG]). Our initial results show encouraging results of correlation between MRI evaluation of dyssynchrony and cardiac resynchronization therapy (CRT) outcomes. In the new proposed research, we will further develop these methods, with the goal of automating the cardiac analysis methods. This will include the introduction of new ML-based methods, which will incorporate information on the specific cardiac motion factors that lead to classification of different disease states in dyssynchrony. Our Hypothesis is that by using these new ML-based methods for cardiac motion analysis, we will discover and evaluate significant quantitative correlations between different cardiac dyssynchrony motion patterns and CRT outcomes. Also, late-gadolinium enhancement (LGE) provides images for infarction visualization. Incorporation of tissue characterization into the motion-pattern analysis could lead to increased understanding of how infarcted areas affect regional motion in concert with dyssynchrony. The unearthing of these findings will allow us to validate them in future clinical studies. The project will also disseminate our novel, coupled DML and model-based methodology for quantifying and classifying cardiac motion in diseases affecting regional wall motion. Other research groups can then apply our tools to specifically study dyssynchrony, as well as other cardiac diseases affecting LV motion.