RISEI Lab Research · Innovation · Elevation Est. 2021 Northwestern University $50M+ Lifetime Funding 170+ Countries Evanston, Illinois RISEI Lab Research · Innovation · Elevation Est. 2021 Northwestern University $50M+ Lifetime Funding 170+ Countries Evanston, Illinois

NAEP Process Data Innovation Grant

Digital Assessment Accessibility and Student Performance

Project Overview

Funding: Institute of Education Sciences (IES)
Lead Institution: Northwestern University RISEI Lab
Assessment Partner: National Assessment of Educational Progress (NAEP)

This research initiative addresses critical gaps in understanding testing accommodations for students with disabilities. The project investigates how accessibility features in digital assessments affect student performance using process data from the National Assessment of Educational Progress (NAEP), examining the relationship between availability and utilization of accessibility features and math performance.

Background & Context

The Challenge

Standardized testing poses significant challenges for students with disabilities, particularly following the No Child Left Behind Act of 2001. During 2017–18, approximately 14% of U.S. K–12 students received Individualized Education Programs (IEPs), with an additional 2.3% qualifying for 504 Accommodation Plans.

The Knowledge Gap

Despite extensive research on testing accommodations, understanding their effectiveness remains limited due to scarce or unreliable utilization data. This project aims to bridge that gap through analysis of digital assessment accessibility features and their impact on student outcomes.

Core Research Questions

The initiative examines several critical questions about accessibility in digital assessments:

  • How do students with disabilities utilize accessibility features in digital assessments?
  • What is the relationship between availability of accessibility features and math performance?
  • How does actual utilization of these features affect test outcomes?
  • Can process data from large-scale assessments identify students who may benefit from extended time or other accommodations?
  • What patterns emerge in mathematical problem-solving when analyzing assessment log data?

Publications & Working Papers

Published Research

Universal by Design: Unveiling the Effectiveness of Accommodations and Universal Design Features through Process Data
Ogut, B., Yin, M., Vu, H., Hicks, J., & Circi, R. (2025). Educational Measurement: Issues and Practice, 44(4), 18-32.

Using NAEP process data with inverse probability weighting and doubly robust models, this study examines how testing accommodations and universal design features affect performance for students with disabilities. Extended time significantly improves outcomes; results for text-to-speech and equation editors are mixed, suggesting these features may sometimes act as distractions rather than supports.

View Publication →
Running out of time: Leveraging process data to identify students who may benefit from extended time
Ogut, B., Circi, R., Huo, H., Hicks, J., & Yin, M. (2025). International Electronic Journal of Elementary Education, 17(2), 253–265.

Using an XGBoost model to analyze 2017 NAEP Grade 8 Mathematics assessment process data, this study identifies students who would benefit from extended time accommodations based on early test behaviors. The model demonstrated high accuracy in predicting the need for extended time with minimal influence from background variables.

72%
Students with disabilities granted extended time did not use it fully
24%
Students lacking extended time were still actively engaged when timed out
High Accuracy
XGBoost model predicts extended time need from early behaviors
View Publication →
Exploring mathematical problem solving through process mining: Insights from large scale assessment log data
Ogut, B., Webb, B., Hicks, J., Circi, R. & Yin, M. (2024). Computers in the Schools, 1–31.

This research applies process mining techniques to large-scale NAEP assessment log data, analyzing student problem-solving strategies on an Algebra problem involving Pascal's triangle. The study identifies distinct patterns that distinguish successful from unsuccessful problem-solving approaches.

3 Strategies
Successful students follow one of three structured problem-solving strategies
Systematic Process
Pascal's triangle pattern reveals structured approach
Predictive Power
Process mining distinguishes effective from ineffective approaches
View Publication →

News & Media

  • New Research on Digital Assessment Accessibility Unveiled by Education Researchers (July 16, 2024)
    Groundbreaking findings on how accessibility features in digital assessments impact student performance, particularly for students with disabilities.
    Read More →

Impact & Applications

This research has important implications for educational policy and practice:

  • Assessment Design: Informing the development of more accessible digital assessment platforms
  • Accommodation Policy: Providing evidence-based guidance for IEP and 504 Plan accommodations
  • Teacher Training: Supporting educators in understanding how students use accessibility features
  • Educational Technology: Guiding the implementation of accessibility features in learning management systems
  • Policy Development: Informing state and federal testing accommodation policies

Project Contact

For more information about the NAEP Process Data Innovation Grant, please contact:
Email: risei@northwestern.edu
Phone: 847-491-7377

Visit RISEI Lab Project Page →