Short Course of 2021 IMPS

Date: July 19, 2021 10:00 am - 5:00pm (EDT)



This full-day short course introduces a selection of statistical learning methods for analyzing process data, that is, log data from computer-based assessments. Covered topics include (1) data-driven methods for extracting features from response processes via n-grams, multidimensional scaling, and sequence-to-sequence autoencoders; (2) introduction to ProcData, an R package for process data analysis; and (3) applications of process features to practical testing and learning problems, including scoring, differential item functioning correction, and computerized adaptive testing. Mode of instruction will be a blend of presentations, for topic (1), and concrete illustrations in R, for topics (2) and (3). Intended audience are researchers and practitioners interested in data-driven methods for analyzing process data from assessments and learning environments. To fully engage in the hands-on activities, familiarity with R and RStudio is expected. Running the ProcData package requires installation of R and Python. Installation instructions and support will be provided. Participants are expected to bring their own laptop with Windows or Mac operating system. By the end of the course, participants are expected to get a composite picture of process data analysis and know how to conduct various analyses using the ProcData package.


Registration is required through 2021 IMPS website.


This short course will be held on July 19, 2021. Below is the course schedule. All times are in Eastern Daylight Time (EDT).

Time Lecturer Session Title
10:00 am — 10:15 am Qiwei He Opening remarks
10:15 am — 11:45 am Jingchen Liu Overview of process data analysis
12:00 pm — 01:30 pm Qiwei He Introduction to n-grams and longest common subsequence
01:45 pm — 03:15 pm Xueying Tang Introduction to ProcData package
03:30 pm — 05:00 pm Susu Zhang Partial scoring and DIF correction


Session 1. Overview of process data analysis

Recent advances in informational technology has led to the increasing popularity of computer-based interactive items, which require test-takers to complete specific tasks within a simulated environment. In addition to the final outcomes, the entire log of interactions between the test-taker and the item, i.e., the sequence of actions and their timestamps, are recorded as process data. Process data contain rich information about test-takers’ problem-solving processes that are not recoverable from the final responses. In this overview, we summarize our main research developments of process data analysis. It includes feature extraction via multidimensional scaling and neural-network-based autoencoder. An important question is how process data can assist specific psychometric research. To address this problem, we present two applications: improving test reliability by constructing a process-data-based partial score system and removing/reducing differential item functioning by including process data in the scoring rules.

Session 2. Introduction to n-grams and longest common subsequence to create process features

We introduce two approaches to handle the sequential process data to derive informative features. The topics covered in this session include

  1. Applying n-grams model to disassemble the long sequences into easy-handled mini-sequences (e.g., unigrams, bigrams and trigrams as examples). We will introduce how to add term weights to n-grams and select the robust n-grams as process features.
  2. Applying longest common subsequence (LCS) to calculate sequence distance. We will demo how to calculate the longest common subsequence between two sequences (e.g., a predefined reference sequence and an individual observed sequence) and derive the variables of sequence similarity and efficiency.

Session 3. Introduction to ProcData

We introduce ProcData, an R package we design for processing, examining, and analyzing process data. The topics covered in this session include

  1. Installation of ProcData
  2. Class proc and its print and summary methods
  3. Climate control dataset cc_data
  4. Reading and writing response processes via read.seqs and write.seqs
  5. Functions for data processing
  6. MDS feature extraction via seq2feature_mds
  7. Autoencoder feature extraction via seq2feature_seq2seq
  8. Fitting sequence models via seqm

We will demonstrate the features of ProcData through live R sessions.

Session 4. Partial scoring and differential item functioning correction via process data

We provide two specific applications of process data analysis to psychometric problems. These two examples illustrate how to make use of the additional information in process data and to what extent they add values to the existing literature.

  1. Accurate assessment of examinees’ abilities is the key task of testing. Traditional assessments are based on the item final responses, while problem-solving processes contain additional information about a student’s proficiency on the measured trait. We establish a framework to systematically construct a process-data-based scoring system that is substantially more accurate than the traditional IRT-model-based assessment in terms of reliability.

  2. Differential item functioning (DIF) can jeopardize test fairness and validity. Various methods have been developed to identify DIF. However, few results are available to reduce or correct DIF. We develop a framework that identifies and further constructs a scoring rule to reduce DIF. This new scoring rule is based on an individualized score adjustment with process data.

In this section, we provide a step-by-step instruction of these two methods through simulated data.

Presenter Information

Qiwei He, Educational Testing Service

Dr. Qiwei (Britt) He is Research Scientist in the Center for Psychometrics and Data Science Modeling at Educational Testing Service (ETS). She holds a PhD of Psychometrics and Data Science from University of Twente, Netherlands. She has research interests in educational and psychological measurement, data/text mining, with specific attention to complex new data source in computer-based tests (e.g., process data, textual data) and methodology advancement in large scale assessments (e.g., PISA, PIAAC). She is the recipient of 2019 Jason Millman Promising Measurement Scholar Award given by the National Council on Measurement in Education (NCME), 2017 Alicia Cascallar NCME Award for an Outstanding Paper by an Early Career Scholar and the OECD Thomas J. Alexander Fellowship. She is leading an OECD project in leveraging process data in dynamic navigation of reading from multiple-source texts and co-leading an NSF-funded project to develop latent and graphical models for complex dependent data in education. Email:

Jingchen Liu, Columbia University

Dr. Jingchen Liu is Professor of Statistics at Columbia University. He holds a Ph.D. in Statistics from Harvard University. He is the recipient of 2018 Early Career Award given by the Psychometric Society, 2013 Tweedie New Researcher Award given by the Institute of Mathematical Statistics, and a recipient of the 2009 Best Publication in Applied Probability Award given by the INFORMS Applied Probability Society. He has research interests in statistics, psychometrics, applied probability, and Monte Carlo methods. He is currently an associate editor of Psychometrika, British Journal of Mathematical and Statistical Psychology, Journal of Applied Probability/Advances in Applied Probability, Extremes, Operations Research Letters, and STAT. Email:

Xueying Tang, University of Arizona

Dr. Xueying Tang is an Assistant Professor in Statistics in the Department of Mathematics at the University of Arizona. Prior to joining the University of Arizona, she was a postdoctoral research scientist at Columbia University in the Department of Statistics. Her research interests include high dimensional Bayesian statistics, latent variable models and their applications in education and psychology. Email:

Susu Zhang, University of Illinois at Urbana-Champaign

Dr. Susu Zhang is an Assistant Professor of Psychology and Statistics at the University of Illinois at Urbana-Champaign. Her research interests include latent variable modeling, the analysis of complex data (e.g., log data) in computer-based educational and psychological assessments, and longitudinal models for learning and interventions. Email: