Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching
Plain Language Summary
Each semester, student evaluations of teaching (SETs) pile up fast, but no one has the time to read through the thousands across university courses to find meaningful patterns. This paper builds a method that uses AI language models to sort large volumes of that feedback into a set of recurring themes, doing automatically what researchers otherwise construct by hand over after much close reading. Applying the method to 5,000 real comments from a public university, we find that the AI-generated themes largely reproduce, and in places refine, a codebook built manually in an earlier study of the same course context. The aim is to make this kind of large-scale feedback actually usable by instructors and administrators rather than left unread.
Contribution
The paper introduces EECS (extract, embed, cluster, summarize), an open-source, locally run pipeline for inductive qualitative codebook generation, distinguishing itself from prior LLM-assisted qualitative coding work, which has largely relied on proprietary models like ChatGPT and been confined to deductive coding against a pre-existing framework.
Research Questions
- Can an automated extract-embed-cluster-summarize workflow, built on open-source generative text models, produce an inductive codebook from a large corpus of student evaluations of teaching that captures themes comparable to those identified through traditional manual thematic analysis?
- How does the resulting codebook compare, in coverage, granularity, and categorical alignment (via the Academic Plan Model), to a codebook generated by human coders on a subset of the same data?
- How does this approach perform relative to other computational methods for thematic analysis (e.g., LDA topic modeling, BERTopic)?
Methods
We applied the seven-step EECS workflow to 5,000 SETs (4672 unique after pre-processing) collected from introductory science and engineering courses at a large public R1 university. The pipeline extracts discrete ideas from each response using the open-source Mistral-7B model, embeds those ideas with the UAE-Angle embedding model, clusters them via HDBSCAN, selects a representative idea per cluster, and generates and then simplifies a codebook through iterative, retrieval-augmented prompting. The resulting codebook was compared against 39 codes from a prior manual thematic analysis of a subset of the same SETs, organized using the Academic Plan Model, and against codes produced by LDA and BERTopic run on the same dataset.
Key Findings
The workflow reduced 12,046 extracted ideas into 272 clusters, an initial codebook of 232 codes, and, after simplification, 80 codes (75 retained after excluding five as ambiguous). These EECS-generated codes overlapped substantially with the 39 human-generated codes across Academic Plan Model categories while also surfacing finer-grained distinctions and one new category, Content and Sequence, that the manual coding had not identified. Relative to LDA and BERTopic on the same corpus, EECS produced more interpretable, semantically coherent labels, though it took considerably longer to run than LDA, which completed in minutes.
Implications
With researcher oversight built in at multiple stages, an open-source LLM-based inductive coding pipeline appears capable of approximating, and in some respects extending, manual thematic analysis at a scale infeasible by hand, while also keeping sensitive institutional data off third-party servers. The authors are careful to frame this as an aid rather than a substitute for human judgment, since generated codes still require pruning and validation, and they flag codebook granularity, dependence on the underlying model, and the context-sensitivity of code interpretation as open problems for future refinement.