The Problem With Prompting: Using Decision Models for Traceable and Justifiable GenAI-Infused Qualitative Data Analysis
Plain Language Summary
Researchers increasingly use AI chatbots built on large language models (LLMs) to help analyze interviews and other text, yet these models are trained to be agreeable rather than accurate, and getting them to reliably mirror human judgments can take months of trial-and-error "prompt engineering." We argue that a newer kind of AI, known as "System One" decision models, fits this work better: rather than writing text, they report how likely each of a fixed set of options is, which lets researchers see, check, and justify each decision the model makes. Drawing on our own study of how engineering instructors describe AI, we recount our move from prompt engineering to these models and what it suggests for qualitative research. The paper is the third step in a line of our work: our review of transcription showed what happens to qualitative traditions when AI quietly changes how research gets done, our "Refutable Reflexivity" model proposed how researchers could make AI-assisted analysis checkable, and here we put that model into practice.
Contribution
We bring System One decision models trained through reinforcement learning for calibrated decisions (RLCD), namely TypeSafe AI's Jev and its open-source counterpart Laya, to GenAI-infused qualitative data analysis (QDA), arguing that their calibrated probabilities over researcher-defined options supply the traceability and answerability that prompt-engineered, RLHF-trained LLMs cannot, while their size, speed, and programmatic operability widen access relative to LLMs; in doing so, we put into practice the refutability criterion we conceptualized in "Refutable Reflexivity: A Theoretical and Visual Model for LLM-Infused Qualitative Research", itself prompted by our finding in "Revisiting Transcription in the Age of AI" that qualitative traditions shift, largely unexamined, when AI changes the work beneath them.
Research Questions
- How might System One decision models trained with RLCD provide the traceability and answerability that prompt-engineered LLMs lack in GenAI-infused QDA?
- How do System One models compare with prompt- and context-engineered open-source LLMs when scaling a manual classification of instructors' figurative language about GenAI?
Methods
We situate this paper as the applied counterpart to two of our earlier works, "Revisiting Transcription in the Age of AI", which traced what happens to qualitative traditions when AI-driven changes go unconsidered, and "Refutable Reflexivity", which conceptualized how LLM-infused research might remain traceable and open to refutation; here we test what LLMs and decision models actually do for GenAI-infused QDA. We present a narrative and experiential case study drawn from an in-progress study of the figurative language engineering instructors use to describe GenAI systems. Building on a pilot in which a subset of instructors' descriptions was classified manually, we recount our attempts to scale that classification, first by prompt- and context-engineering open-source LLMs over several months and then, following their release in September 2026, with System One models. We pair this account with an argument contrasting how RLHF-trained LLMs and RLCD-trained decision models arrive at their outputs, and what each affords qualitative inquiry.