Can an AI Judge Your Exam? What 2026 Research Reveals About LLMs as Graders

The Grading Crisis That Won't Go Away

India's exam evaluation system operates at a scale that is genuinely difficult to comprehend. Each year, state boards and central bodies like CBSE, CISCE, and NTA collectively process hundreds of millions of answer scripts. During peak evaluation season — February through May — universities and boards scramble to recruit qualified evaluators, manage evaluation centers, and complete marking within tight result declaration windows.

The pressures of this scale create predictable problems: inter-examiner variability, fatigue-related errors, disputes that generate tens of thousands of revaluation applications, and systemic delays that ripple into college admissions timelines. For years, on-screen digital evaluation has addressed the workflow and accuracy problems. But the question that researchers, vendors, and education technologists have increasingly turned to is more provocative: can a large language model (LLM) do the grading itself?

In 2026, a cluster of peer-reviewed studies has attempted to answer that question with rigor. The findings are nuanced, instructive — and more immediately relevant to Indian exam administrators than the headlines suggest.

What Is LLM-as-Judge?

The "LLM-as-judge" framework refers to using large language models — the same category of AI that powers ChatGPT, Claude, Gemini, and their peers — as automated raters for assessment tasks. Rather than relying on keyword-matching or simple pattern recognition, these models read the full text of a student's response, interpret it against a rubric, and assign a score.

The key difference from earlier automated scoring systems is the depth of language understanding. LLMs can process essays, long-answer descriptive responses, and context-dependent arguments in a way that earlier natural language processing tools could not. Proponents argue this makes them suitable for the kinds of subjective, open-ended questions that dominate university examinations in humanities, social sciences, and professional programs.

What the 2026 Research Found

A study published in the Journal of Artificial Intelligence and Technology evaluated multiple leading LLMs — including ChatGPT-5, Gemini Advanced 2.5 Pro, Qwen-3 Max, Mistral Le Chat Pro, and a locally fine-tuned LLaMA 3.3 70B — on authentic midterm exam scripts from language courses. The performance on subjective grading tasks varied significantly:

Human graders, supported by structured scoring rubrics, remained "nearly perfect" in terms of accuracy and inter-rater reliability. The gap between the best LLM and a well-trained human examiner was roughly 15 percentage points.

A separate 2026 study on grading scale design found that LLMs achieve highest alignment with human judgments when using a 0–5 scoring scale rather than wider scales. On 0–10 or percentage-based marking, the agreement deteriorates — a finding that has direct implications for how any AI-assisted grading tool would need to be calibrated for Indian examination systems, which typically use 100-point or subject-specific scales.

Research from Cornell and Carnegie Mellon specifically examined LLMs deployed for grading and appeal resolution. One striking data point: the appeal process led to grade changes in 74% of cases where students contested an AI-assigned grade. This suggests that while LLMs can grade at speed, their outputs require robust human review mechanisms before they can be used in consequential assessments.

Where LLMs Perform Well

The research consistently identifies conditions under which LLM grading is reliable:

Structured rubrics significantly close the gap. When evaluators provide detailed, hierarchical rubrics — specifying exactly what constitutes each mark level — LLMs achieve what one study calls "human-comparable consistency" even on subjective dimensions. Vague marking schemes, by contrast, allow both human and AI inconsistency to compound.

Objective-like judgments within subjective responses. For questions that have a partially determinable answer — whether a student correctly identified key concepts, whether required steps in a problem are present — LLMs perform at or near human accuracy. The unreliability concentrates in purely aesthetic or interpretive dimensions: writing style, argument quality, and creative expression.

Formative assessment and feedback. Multiple studies found that students valued the speed and detail of AI-generated feedback even when they questioned the fairness of the grade itself. For internal assessments where feedback quality matters more than the exact mark, LLM graders offer clear practical value.

Where They Fall Short

The limitations are equally consistent across studies.

High inter-scale disagreement on open-ended quality benchmarks. When questions require genuine interpretive judgment — evaluating the quality of an argument, assessing historical analysis, or grading a creative piece — LLMs show substantially lower reliability than on factual or structured responses.

Language and script diversity. India has 22 official languages and hundreds of regional dialects. Most state board answer scripts are handwritten, and a significant proportion are in regional languages and scripts. Current LLMs are overwhelmingly trained on English-dominant datasets and perform poorly on handwritten regional-language text, even after OCR conversion. The linguistic diversity of Indian examinations represents a structural obstacle that no commercially available LLM has yet adequately addressed.

Absolute standard judgments vs. relative ranking. LLMs are better at ranking answers relative to each other than at assigning absolute marks against a fixed standard. Indian board examinations require absolute assessment — a student must earn 33 marks out of 100 to pass, not merely score above average. This distinction matters enormously for consequential decisions like pass/fail determinations.

Implications for Indian Digital Evaluation

The research does not support the conclusion that LLMs are ready to replace human evaluators in India's examination system. It does support a more targeted and immediate conclusion: LLMs can meaningfully assist human evaluators within a well-designed digital evaluation platform.

The most credible near-term applications are:

The Human-in-the-Loop Imperative

The 74% appeal change rate from the Cornell/Carnegie Mellon research is a cautionary figure. It suggests that AI grading without meaningful human oversight produces outcomes that students — and likely courts — would not accept as legitimate. For India's high-stakes board examinations, where a single mark can determine college admission, the consequences of an unchecked AI grading error are disproportionate.

The digital evaluation platforms that are gaining adoption across Indian universities and boards — including CBSE's on-screen marking rollout for Class 12 in 2026 — are precisely the human-in-the-loop systems that align with the research findings. The evaluator remains the decision-maker; technology structures and supports that decision.

The Road Ahead

The research trajectory is optimistic. Accuracy is improving rapidly — the 15-percentage-point gap between best LLM and human grader is narrower than equivalent measurements from 2024. Fine-tuned models trained on domain-specific examination rubrics and regional language text outperform general-purpose models significantly.

India's PARAKH (Performance Assessment, Review, and Analysis of Knowledge for Holistic Development) body, established under NEP 2020 to reform assessment standards, has an explicit mandate to evaluate emerging technologies for examination use. As rubric standardization improves and as multilingual LLMs become more capable, the case for limited AI-assisted grading in Indian examinations will strengthen.

For now, the responsible use of this technology is as a quality-control layer and efficiency tool within platforms that keep qualified human evaluators at the centre of every consequential grading decision.

Related Reading