Brian W. Junker - US grants

The focus of this work is on strictly unidimensional latent variable models, which may be thought of as mixture models which induce conditional association (Rosenbaum, 1984; Holland and Rosenbaum, 1986) in the marginal distribution of the data, or as a special case of Stout's (1987, 1990) essentially unidimensional models. I propose to extend my past efforts to use these ideas together with notions from the literature on positive and negative dependence (e.g. Joag-Dev, 1983; Joag-Dev and Proschan, 1982; Newman and Wright, 1981; or more recently the collection edited by Block, Sampson and Savits, 1990) to characterize strictly unidimensional models. My recent explorations of this problem suggest a reasonably straightforward approach that, as a side benefit, generalizes de Finetti's characterization of exchangeability, without the need to specify sufficient statistics as in, for example, Diaconis and Freedman (1984). A second line of work in this proposal is the exploration, using asymptotic methods along the lines of Kass, Tierney and Kadane (1990), and Clarke and Barron (1990), of inferences about the latent trait under a strictly unidimensional model, which asserts conditional independence given the latent trait, when in fact some mild form of conditional dependence holds. In addition, biases in the asymptotic standard error of an MLE-like estimator can also be calculated and, in some cases, corrected using nonparametric regression ideas due to Ramsay (Ramsay, 1991; Ramsay and Winsburg, 1991). Finally, some problems in applications and computing will be examined, including unifying and extending nonparametric techniques for latent variables data analysis (e.g. Molenaar, 1991; and Grayson, 1988); and developing parametric statistical models and computational methods (e.g. efficient estimation of a polytomous version of the model specified by Lindsay, Clogg and Grego, 1991) that arise in the analysis of data from small scale experiments in cognitive science. This proposal concerns statistical and probabilistic features of latent variable models for repeated measures data, which is of interest to quantitative psychologists, psychometricians, and cognitive scientists, as well as other social scientists. A typical application for latent variable models is psychological measurement, in which the latent variable is an unobservable variable that indicates the level of a psychological feature of a person---such as depression, mathematical aptitude, job satisfaction, or working memory capacity---that we observe only indirectly through the person's responses to a series of tasks, questionnaire items, etc. Data of this type might be obtained from psychiatric rating forms, standardized academic achievement or aptitude tests like the SAT and GRE, standardized questionnaires in sociology, or coded responses to a set of tasks in experiments in cognitive psychology. A primary outcome of this research will be a deeper understanding of latent variable models for measurement problems, at both the level of fundamental statistical theory and the level of practical applications. Practical tools arising from this research would include: enhanced methods for deciding how well or poorly this class of models matches particular situations or data sets; rules for adjusting scientific inferences based on these models for the inevitable mismatch, however small, between the model being used and the mechanism that generated the data; and computational and model-building methods that are adapted to small-scale experimental data, such as might be found in cognitive psychology, where these models are conceptually natural but current methods tend to break down. Much of the work proposed here is built around interdisciplinary collaboration, especially with quantitative psychologists and educational measurement specialists, with the goal of developing statistical theory that will be of use in applications.

NSF DMS-9705032 Latent variable models in action: hierarchical Bayes and mixture models for repeated discrete measures with individual differences Brian Junker Carnegie Mellon University PROJECT ABSTRACT: A central feature of this research is the development of widely applicable methodology for latent variable models for measurement problems in education, psychology and the social sciences. This methodology is being developed and tested in several specific areas: Monotonicity and stochastic ordering properties that follow from the strictly unidimensional latent variable representation are being studied and applied to nonparametric scaling problems. A promising Markov chain Monte Carlo method is being extended and applied to a variety of problems, including: correct modeling of rater variability in educational achievement data; accomodating heterogeneous catchability in multiple-recapture censuses; and developing methods for multidimensional and hierarchical latent variable models for discrete repeated measures. In addition, the research addresses the sensitivity of inferences to underspecification of the model. A second thrust of the research is to refine and develop existing characterizations of unidimensional latent structure into a statistical theory of, and statistical methods for assessing, latent variable dimensionality. This work aims to more fully blend psychometric and statistical approaches to latent variable models for repeated discrete measures. Psychometric methodology tends to concentrate on model building and model features; and psychometric data analysis tends toward issues of scaling (selecting questions that ``hang together'' in the sense that a unidimensional latent variable model holds), reliability (ensuring that the latent variable can be estimated well from the questions selected), and the assessment of latent variable dimensionality from data. Statistical methodology tends to sidestep these bas ic psychometric questions, and instead concentrates on finer model adjustments, and various inferential and predictive tasks. The focus of this research is on statistical and psychometric features of latent variable models for repeated measures data, which is of interest to quantitative psychologists, educational measurement specialists, and cognitive scientists, as well as other social scientists. Much of the work is collaborative in nature, and it is built around the development of theory and methodology motivated from, and useful for, substantive applications. --------------------------------------------------------------------------

9720354 Lovett This project is being funded by the Learning and Intelligent Systems (LIS) Initiative, including support from the Office of Multidisciplinary Activities of the Directorate for Mathematics and Physical Sciences. This project will develop the three core components of an innovative, intelligent learning environment for teaching statistical reasoning. It is aimed at directly facilitating students' ability to transfer what they have learned to situations outside the original learning context. The three components are (1) a computer interface that helps students develop a general understanding, (2) a detailed specification of the knowledge required to apply statistical reasoning effectively, and (3) new computational and statistical techniques for assessing the accuracy and generality of students' knowledge and then generating appropriate remediation. This project entails a unique collaboration among cognitive psychologists, statisticians, and computer scientists. This project will lead to fundamental advances on several fronts. First, the interface provides a new learning tool that will be used by every humanities and social sciences student at Carnegie Mellon University and will be disseminated to other colleges. Second, because the interface is designed to apply the principles revealed by recent cognitive psychology research, it offers a test of these principles' effectiveness in practice. Third, developing a detailed specification of the knowledge required for statistical reasoning will yield new insights that can inform statistics instruction and cognitive theories. Fourth, the techniques for assessing students' knowledge develop new ways of using the information recorded by computerized learning environments. Fifth, the rich data collected on students' transfer throughout this project will lead to a deeper understanding of how, when, and why transfer occurs. Statistical reasoning is the domain for this project because (a) effective transfer is critical here--stude nts must apply the skills they have learned across a wide range of issues and content areas, and (b) students often have great difficulty transferring these skills.

This project will investigate an innovative, technology-enabled approach to elementary reading - an automated reading tutor that displays stories on a computer screen, listens to children read aloud, and helps them learn to read. Experimental use in an urban elementary school has provided initial evidence of the Reading Tutor's ability to engage students in sustained assisted reading. In a four-month controlled study, children who used the Reading Tutor gained significantly more in reading comprehension than classmates who spent the same time in regular activities.

The main question addressed by the project is the efficacy of the Reading Tutor when used in real classrooms. The experiments include an evaluation of the Reading Tutor's longitudinal effects on student progress, compared to commercial software and to human tutors. Expected outcomes include demonstrating the effectiveness of speech-recognition-enabled assistance in building and assessing reading skills, and establishing how much various type so learners, problems, and skill sets benefit from such assistance.

Sustainability and scalability will be addressed by analyzing and addressing key factors at the interface between technology and the classroom that affect the amount of usage of the system. Experiments with successive prototypes of a Reading Tutor that runs on commercially available computers will lay research foundations for transition of it and other educational software into affordable, effective, scalable interventions that can make a real difference in children's education.

This award supports the investigator's work at the University of Pittsburgh's Learning Research and Development Center (LRDC). Projects to be initiated include: (1) Analyzing school district data archives with an eye toward evaluating educational progress and monitoring the outcomes of educational innovations; (2) Exploring social judgement in education, in particular in the development of an institutional portfolio rating system for classrooms and schools, based on the "Principles for Learning" of LRDC's Institute for Learning; and (3) Laying technical groundwork for a bank of linked topical tests instantiating a purely standards-referenced testing program. All three are connected to ongoing research programs at LRDC. These projects are designed to contribute to the development of data collection systems for adequate school accountability systems and for educational policy evaluation. Research conducted through the Institute for Learning and elsewhere suggests that sustained improvement in student achievement is most reliably attained through institutional change. Yet most currently implemented accountability systems focus instead on individual student outcomes, and often are confounded with high-stakes decisions for individual students. The first project will explore whether existing school district data archives can be exploited to limit additional individual student testing when student achievement data is called for. The second project will apply methodology developed over the past ten years for student portfolio assessment to the development and rating of institutional portfolios intended to show that local institutions (e.g., classrooms, schools and districts) are engaged in a process of professional development that ensures long term gains for students. The banked tests in the third project would each cover fairly narrow topics, such as integer arithmetic, fractions, etc., and could be used for example to assess the distribution of student achievement within a district, school, or classroom relative to specific learning standards. This research is supported by the Methodology, Measurement, and Statistics Program and the Statistics and Probability Program under the Mid-Career Methodological Opportunities Fellowship Announcement.

9819900
Eddy

At Carnegie Mellon University, the Department of Statistics and the Department of Mathematical Sciences will build on their complementary strengths to develop a joint, vertically-integrated program of education and research. Responding to national needs, the program will (i) train postdoctoral fellows for careers emphasizing research in settings that require versatility, (ii) aim to recruit and retain U.S. graduate students, avoiding excessive time to complete Ph.D.s while providing students with a high probability of success after graduation, and (iii) help increase the numbers of U.S. undergraduates, including women and minorities, who pursue advanced degrees in mathematical and statistical sciences. The program emphasizes cross-disciplinary research and understanding the needs of learners in a context of disciplinary advancement. Many of the activities grow from two previously-funded Group Infrastructure Grants to our respective departments, and from a very successful Undergraduate Summer Research Institute in Applied Mathematics. For instance, we plan to use the graduate support model from the infrastructure grant to Mathematical Sciences, we will expand the operation of the Summer Institute to include students from Statistics, and we will adapt for Mathematical Sciences some of the postdoctoral mentoring procedures that have worked well in Statistics.

Our evaluation of this training program will assess the following:
involvement of undergraduates in meaningful research experiences; its success in producing acceptable average time-to-degree for VIGRE graduate trainees; its effectiveness in expanding the mathematical horizons and career opportunities of students at both the undergraduate and the graduate levels, with particular focus on the graduate program; its effectiveness at the postdoctoral level in preparing VIGRE postdoctoral fellows for careers as professional mathematical scientists; its effectiveness in developing the communications skills of VIGRE participants; the effectiveness of the mentoring of undergraduate students, graduate trainees, and postdoctoral fellows; overall effectiveness of the research teams and other efforts to integrate research and education; the effectiveness of the partnership-in-training between the Departments of Statistics and Mathematical Sciences; and the degree of involvement of women and minorities.

Funding for this award is provided by the Division of Mathematical Sciences and the MPS Directorate's Office of Multidisciplinary Activities.

Failure to learn hidden skills is a persistent obstacle to students in science, math, and engineering domains. Hidden skills, which include problem categorization, feature detection, and planning, are critical to solving problems in a domain but do not have any immediate, external product for students to see. Unfortunately, it is unclear how best to identify and teach these difficult-to-learn skills. Instructional scaffolding is a popular and effective technique for providing targeted support and guidance while students learn to solve problems in a new domain. Scaffolding has great potential for improving hidden-skill learning. However, the reasons it works and how best to implement it are largely unknown.

The proposed research will explain the effectiveness of instructional scaffolding in terms of hidden skill learning. Several hypotheses about the relationship between scaffolding and hidden skills will be tested, and new scaffolding designs will be evaluated. This will lead to a systematic approach to teaching hidden skills that improves students' learning and transfer. The four specific aims of this project are:
(1) Develop a systematic, efficient method for identifying hidden skills. While methods currently exist for analyzing domain-specific knowledge, these methods are not robust for identifying hidden skills, and they tend to be difficult and slow. This project will develop and test an automated method that combines logistic regression models and heuristic search algorithms to infer where hidden skills lie.
(2) Develop a theoretical explanation for why scaffolding works. Although instructional scaffolds often lead to better learning, there has been little theoretical progress in explaining when and how scaffolding works. A sequence of experiments will be conducted to test three hypotheses that offer increasingly concrete levels of explanation for how scaffolding benefits learning and transfer.
(3) Develop practical guidelines for the design of effective instructional scaffolding. Three critical questions for scaffolding design will be examined: What level of scaffolding support is sufficient to achieve its main benefit? When and how should scaffolding support be built and faded? And how can human instructors (i.e., TA's) best complement a computerized scaffolded learning environment?
(4) Develop novel applications of our results on scaffolding hidden skills. There are at least two novel applications of this work, beyond the scope of learning theory and instructional design. First, the scaffolding designs from Specific Aim 3 will be used to develop new on-line assessments of students' understanding. Second, the results from Specific Aim 1 will be used to develop tools that train instructors to "see" the hidden skills in complex problems and thus better anticipate students' learning difficulties.

At Carnegie Mellon University, the VIGRE program of the Department of Statistics will involve all trainees in supervised, cross-disciplinary research, where they will learn how to translate a research question into well-posed statistical problems, solve these problems, and translate the results back into a product that is accessible to the relevant scientific community. This skill is also central to learning basic statistics and forms a conceptual link between research and education, facilitating their integration. At the graduate level experience in the process includes a year-long project, typically with a faculty member in another domain, while a Statistics faculty member serves as advisor; provides a series of steps to improve communication skills and teaching effectiveness, and mentors in the area of professional growth. The graduate curriculum will be modified to make it more effective in building cross-disciplinary skills. Undergraduates will have several new courses available and will be involved in a capstone research project, and we will run a summer program, emphasizing minority students. Postdoctoral fellows will be involved in research projects, and will co-teach courses with senior faculty. They will also participate in structured mentoring sessions.

Undergraduate, graduate, and postdoctoral trainees will be integrated in research teams. The Carnegie Mellon VIGRE program in Statistics will (i) train postdoctoral fellows for careers emphasizing research in settings that require versatility, (ii) recruit and retain U.S. graduate students, avoiding excessive time to complete Ph.D.s while providing students with a high probability of success after graduation, and (iii) help increase the numbers of U.S. undergraduates, including women and minorities, with advanced training in statistical science. While maintaining a strong disciplinary foundation for statistical practice, the program emphasizes cross-disciplinary research and understanding the needs of statistical novices.

This project deals with the development, testing, and deployment of models for multiple social networks, particularly those with conditionally independent ties. The project will explore their properties with respect to partial pooling across networks, a case that includes a single network or ensemble of networks observed over time. This research is motivated by problems in many sociological fields, particularly education research, in which multiple groups of people form their own networks, including students and teachers alike. The project will build on preliminary constructions of the Hierarchical Latent Space Model and the Hierarchical Mixed-Membership Stochastic Block Model by focusing on how information can be pooled across networks, through hierarchical structure specification, and how model parameters evolve through time, through model-dependent autoregression or other smoothing methods. Multiple ways in which an intervention can affect a subset of these networks also will be studied. These models will use both simulated and real-world data to validate their effectiveness. Standard methods for fitting these models, such as Markov Chain Monte Carlo, will be used initially, though wider deployment of these models will demand the development of quicker inferential procedures based on Variational Inference and/or Sequential Monte Carlo. Finally, model validation will be considered in each of these cases, in terms of comparison to other models as well as the adequacy of a model's fit to data.


Data on multiple social networks arising from the same generative mechanisms, and evolving over time together, are becoming increasingly available in education research, public health, and the social sciences. Instead of treating each network separately or assuming that all come from exactly the same model (which is possible only in limited circumstances), this project will provide a new, clearly formulated methodology to deal with this type of data. Researchers in related fields will have the opportunity to use the methods on their own research. Computer code for these routines also will be made available.

Assessment of individuals' proficiency at complex tasks is often accomplished by observation and rating. Teachers or testing agencies, for example, rate students' essays and their solutions to complex problems in mathematics and science. School districts employ trained observers to rate teachers' performance in the classroom. Experts rate radiologists' ability to classify x-ray images. Ratings, however, may change over time due to changes in the way the rater perceives the work and/or changes in individuals' proficiency. The material being rated also may reflect more than one dimension of proficiency. Finally, summaries of these ratings may be misleading when the data collection design includes groupings (schools, hospitals, etc.) that introduce extraneous statistical dependence into the rating data. This project will expand the Hierarchical Rater Model (HRM), a multilevel item response theory model that accounts for dependencies between multiple ratings of the same work, into a framework that will accommodate (a) variation in ratings over time; (b) multidimensional assessments; and (c) clusters and other hierarchical structure introduced by the data collection design. This new framework will allow the HRM to provide estimates of the overall proficiencies of individuals on the rated tasks, as well as estimates of precision, accuracy, and other rater characteristics, under a broad variety of practical rating situations. Analytical work, simulation studies, and real data applications will be used to explore and demonstrate the feasibility and applicability of the expanded HRM framework. In particular, planned analysis of data from the Measures of Effective Teaching project (MET; Bill and Melinda Gates Foundation, 2012), a large study of class-room teaching in the United States, will demonstrate feasibility of the proposed methodological advancements to the HRM. The research will culminate with a new HRM framework with unified notation and formulations so that researchers may specify and estimate special cases of the generalized model as needed. The project also will provide computational tools including algorithms and source code, so that researchers can apply the framework with ease.

The new HRM framework will advance scientific and practical knowledge in two ways. It will enable researchers and practitioners to obtain high-quality estimates of proficiency that account and adjust for complex structure in the ratings. It also will provide rich information about raters and the rating process. Ratings of work, performance, and behavior are an increasing part of high-stakes decisions in many fields including human resources, medical diagnosis, and psychology. The largest impact of this project may be in education policy and research, where ratings of teachers and students are increasingly common. The new HRM framework will allow researchers and practitioners in these fields to produce more accurate assessments of individuals being rated, and to diagnose possible issues in the measurement and rating design, contributing to improved high-stakes decision making based on rating data.