Publication Detail
Peer-Reviewed Research on Assessment Accessibility
Leveraging NAEP Process Data for Assessment Accessibility & Problem-Solving
Papers in This Series
Paper 1: Universal by Design
Ogut, B., Yin, M., Vu, H., Hicks, J., & Circi, R. (2025). Universal by design: Leveraging NAEP process data to inform assessment accessibility for students with disabilities. Educational Measurement: Issues and Practice, 44(4), 18–32. DOI: 10.1111/emip.70007
Paper 2: Running Out of Time
Ogut, B., Circi, R., Huo, H., Hicks, J., & Yin, M. (2025). Running out of time: Examining extended time accommodations for students with disabilities using NAEP process data. International Electronic Journal of Elementary Education, 17(2), 253–265.
Paper 3: Exploring Mathematical Problem Solving
Ogut, B., Webb, B., Hicks, J., Circi, R., & Yin, M. (2024). Exploring mathematical problem solving through process data. Computers in the Schools. DOI: 10.1080/07380569.2024.2416422
Overview
This series of studies leverages process data from the National Assessment of Educational Progress (NAEP) to investigate how students with disabilities interact with digital assessments. The research examines the effectiveness of testing accommodations, including extended time and text-to-speech, and uses process mining techniques to reveal distinct mathematical problem-solving strategies. Together, these papers provide evidence-based insights for improving assessment design and accommodation policies to better serve all learners.
Key Findings
- Extended time improves performance for SWD: Students with disabilities who received extended time accommodations showed improved performance on NAEP mathematics items, supporting the validity of this accommodation.
- Text-to-speech yields mixed results: The effectiveness of text-to-speech accommodations varied across item types and student populations, suggesting the need for more nuanced accommodation assignment policies.
- 72% of SWD granted extended time didn't use it fully: A striking majority of students with disabilities who were granted extended time did not exhaust their allotted time, raising questions about how extended time functions as an accommodation.
- 24% still actively working when timed out: Among students without extended time, nearly a quarter were still actively engaged with items when standard time expired, suggesting some students may need more time than currently allotted.
- Three distinct problem-solving strategies identified: Process mining analysis revealed three distinct patterns of mathematical problem-solving behavior, providing insights into how students approach and navigate digital assessment items.
Extended Time Accommodation Findings
Methods
Data Source
Process data from the National Assessment of Educational Progress (NAEP), including timestamped student interactions with digital assessment items.
Analytical Approaches
The studies employ a combination of statistical modeling (regression analysis with accommodation indicators), survival analysis (time-to-completion modeling), and process mining techniques to analyze student behavior patterns during assessment.
Populations
Students with and without disabilities participating in NAEP mathematics assessments, with focused analyses on students receiving extended time and text-to-speech accommodations.
Policy Implications
- Rethink extended time allocation. The finding that most SWD do not use their full extended time suggests that accommodation policies should focus on ensuring time sufficiency rather than simply granting additional minutes.
- Address the timed-out population. With 24% of standard-time students still active when time expires, assessment programs should consider whether current time limits are adequate for all test-takers.
- Refine text-to-speech implementation. Mixed results for TTS accommodations indicate the need for research-driven improvements in how text-to-speech is deployed across different item types.
- Leverage process data for universal design. Process data can inform the design of assessments that are more accessible to all students from the outset, reducing the need for individual accommodations.
- Personalize assessment experiences. The identification of distinct problem-solving strategies suggests that adaptive assessment designs could better capture what students know and can do.
Citations
Ogut, B., Yin, M., Vu, H., Hicks, J., & Circi, R. (2025). Universal by design: Leveraging NAEP process data to inform assessment accessibility for students with disabilities. Educational Measurement: Issues and Practice, 44(4), 18–32. https://doi.org/10.1111/emip.70007
Ogut, B., Circi, R., Huo, H., Hicks, J., & Yin, M. (2025). Running out of time: Examining extended time accommodations for students with disabilities using NAEP process data. International Electronic Journal of Elementary Education, 17(2), 253–265.
Ogut, B., Webb, B., Hicks, J., Circi, R., & Yin, M. (2024). Exploring mathematical problem solving through process data. Computers in the Schools. https://doi.org/10.1080/07380569.2024.2416422