EXPLAINABLE AI IN LEGAL REASONING
From Statute Prediction to Sycophancy Detection
"Law aspires to reason.
AI aspires to pattern.
This track asks if they can meet."
AI and Law have rapidly grown as an area of research worldwide, across the judiciary. Researchers across multiple legal systems, tasks, and languages have explored the applicability of LLMs to this field.
Nonetheless, the AI prediction/recommendation system is perhaps inconsequential unless the legal experts find it useful, and, more importantly, interpretable and reliable.
Our track provides a testbed for evaluating the efficacy of LLMs in generating trustworthy and robust solutions to important legal problems — where the stakes are not abstract benchmarks, but the integrity of legal reasoning itself.
Two Tasks
Explainable Statute Prediction
Given the factual description of an Indian Supreme Court case, predict which IPC sections apply, locate the exact triggering sentences, and explain the legal reasoning.
Sycophancy Detection
Detect sycophantic behavior in LLMs — the tendency to echo user beliefs regardless of truth. Predict whether the model will agree or disagree with the user.
Why It Matters
Legal AI must be both accurate and explainable. Our metrics measure not just what the model predicts, but why — grounded in the facts that matter.
* Metric weights are tentative and may be updated before test data release.
Macro F1
Exact match on predicted section labels vs. gold standard — measures whether the model identifies the correct statutes.
ROUGE-L
Longest common subsequence similarity between predicted and gold reasoning — evaluates the quality of explanation.
BLEU
Sentence-level BLEU score between reasoning texts — captures lexical fluency of legal explanations.
Recall@3
Whether gold section labels appear in the top-3 predictions — measures recall under uncertainty.
Legal Semantic Score
Cosine similarity of reasoning embeddings from a legal-domain model — captures deep semantic understanding of legal reasoning.
Timeline
Organizers
Register Now
Participate in the shared task at the intersection of AI and Law. Open to researchers, practitioners, and students worldwide.
Registrations open 15 June 2026