Toward Reflexive LLM-Infused Research: A visual guide for qualitative researchers using LLMs
Plain Language Summary
Researchers who study people through interviews, observation, and open-ended writing increasingly reach for AI language models somewhere in their work, but a sentence saying "we used AI" tells a reader almost nothing about what actually happened. This paper builds a shared vocabulary, and a set of diagrams to carry it, for describing that use precisely: separating the underlying model from the software wrapped around it, and locating each moment a researcher turns to an AI within the looping, untidy course of a real project. We then argue that existing disclosure checklists, useful as they are, do not by themselves make a study's conclusions traceable back to the evidence and judgments that produced them. The aim is to give qualitative researchers a way to show their work with these systems, and to give reviewers a way to scrutinize it.
Contribution
We contribute a visual, comics-informed vocabulary for what we term "LLM-infused" qualitative research — distinguishing "technical" from "configured" LLMs and decomposing a project into "phase-instances," "phase-stacks," and a "project stack" bounded by Preparation and Synthesis — and use it to map where the GUIDE-LLM quality checklist lands and where it stops short, proposing refutability and recoverability as the reflexive criterion that quality disclosure alone cannot supply.
Implications
Because researchers cannot open the full interpretive chain of an LLM-infused project for scrutiny, they must exercise selective judgment about which phase-instances to describe, a representational problem closer to selecting quotations for a findings section than to completing a compliance checklist. We therefore suggest that manuscripts carry both quality disclosures, for which GUIDE-LLM serves as a workable guide, and reflexive disclosures that keep the stack's refutability open, since quality and trustworthiness are not the same thing and complete certainty in LLM-infused qualitative data analysis is not on offer. Handled this way, the field's methodological drift can at least follow the disciplinary trajectories researchers intend rather than those the pace of model releases imposes on them.