TheoremExplainAgent：面向LLM定理理解的多模态解释

原文链接

♠️University of Waterloo, ♡Votee AI, †Vector Institute [email protected], [email protected], [email protected]

Your browser does not support the video tag.

Trailer video for TheoremExplainAgent **Please turn on the audio for the best experience.**

Abstract

Understanding domain-specific theorems often requires more than just text-based reasoning; effective communication through structured visual explanations is crucial for deeper comprehension. While large language models (LLMs) demonstrate strong performance in text-based theorem reasoning, their ability to generate coherent and pedagogically meaningful visual explanations remains an open challenge. In this work, we introduce TheoremExplainAgent, an agentic approach for generating long-form theorem explanation videos (over 5 minutes) using Manim animations. To systematically evaluate multimodal theorem explanations, we propose TheoremExplainBench, a benchmark covering 240 theorems across multiple STEM disciplines, along with 5 automated evaluation metrics. Our results reveal that agentic planning is essential for generating detailed long-form videos, and the o3-mini agent achieves a success rate of 93.8% and an overall score of 0.77. However, our quantitative and qualitative studies show that most of the videos produced exhibit minor issues with visual element layout. Furthermore, multimodal explanations expose deeper reasoning flaws that text-based explanations fail to reveal, highlighting the importance of multimodal explanations.

Figure 1: We do not have knowledge of a thing until we have grasped its cause (Aristotle, 1901). A strong reasoning model should not only generate correct conclusions but also communicate them effectively. Visualization enhances human intuition by making abstract concepts more concrete and revealing hidden relationships. Moreover, visual explanations expose reasoning errors more clearly than text, making it easier to diagnose model mistakes.

How TheoremExplainAgent works?

We introduce TheoremExplainAgent (TEA): a novel agentic system designed to generate explanatory videos of mathematical and scientific theorems. TEA employs a two-agent architecture: a planner agent that creates coherent story plans and narrations, and a coding agent that generates Python animation scripts with Manim. This allows the system to produce long, coherent, and pedagogically meaningful videos capable of effectively communicating complex concepts across various STEM disciplines, exposing deeper reasoning flaws that text-based evaluations often miss. To evaluate the efficacy of AI-generated explanations, and specifically those produced by TEA, we introduce TheoremExplainBench (TEB): a benchmark suite comprising 240 meticulously selected theorems. TEB assesses explanations on five key dimensions: accuracy, depth, logical flow, visual relevance, and element layout, ensuring a comprehensive evaluation of pedagogical soundness. We believe TEA, coupled with the rigorous evaluation provided by TEB, will drive advancements in AI systems capable of generating truly insightful and educational explanations.

Figure 2: TheoremExplainAgent consists of two LLM agents. Taking a theorem as input, the planner agent create plans for execution. The coding agent then generates Python scripts to produce visuals and audio.

Experimental Results

Figure 3: An overview of the multimodal theorem explanation framework.

Based on our proposed benchmark evaluation metrics, we conducted evaluations on different large language models (LLMs) to assess their respective capabilities. The evaluations were performed using both pure inferencing and retrieval-augmented generation (RAG) methods. Below are the detailed results from our experiments:

Table 1: Agent Success Rate in Generating Complete Videos

Table 2: Performance of Proposed Metrics on Successfully Generated Videos

Rendered Theorem Explanation Videos Example (Sound On 🔊)

\

High Quality Videos

Physics - Geometric Brownian Motion

Computer Science - Gradient Descent

Math - Properties of Kites

Math - Rational Zero Theorem

Math - Integration By Substitution

Chemistry - Kjeldahl Method

Physics - Geometric Brownian Motion

Computer Science - Gradient Descent

Math - Properties of Kites

Math - Rational Zero Theorem

Math - Integration By Substitution

Chemistry - Kjeldahl Method

Physics - Geometric Brownian Motion

Computer Science - Gradient Descent

Math - Properties of Kites

Math - Rational Zero Theorem

Math - Integration By Substitution

Chemistry - Kjeldahl Method

Physics - Geometric Brownian Motion

\

Poor Quality Videos

Physics - Electromagnetic Spectrum

Computer Science - K Means Clustering

Physics - Virial Theorem

Chemistry - Hard Acid Soft Base Theory

Math - Pythagorean Theorem

Chemistry - Michaelis Menten Kinetics

Physics - Electromagnetic Spectrum

Computer Science - K Means Clustering

Physics - Virial Theorem

Chemistry - Hard Acid Soft Base Theory

Math - Pythagorean Theorem

Chemistry - Michaelis Menten Kinetics

Physics - Electromagnetic Spectrum

Computer Science - K Means Clustering

Physics - Virial Theorem

Chemistry - Hard Acid Soft Base Theory

Math - Pythagorean Theorem

Chemistry - Michaelis Menten Kinetics

Physics - Electromagnetic Spectrum

Case Study

A key finding is that visual explanations significantly improve error diagnosis compared to text-based explanations. While text can reveal that an error exists (as seen in Figure 4), it often fails to elucidate why the error occurred. For instance, a text explanation might indicate incorrect application of the chain code theorem without pinpointing the underlying flaw in reasoning. In contrast, video-based explanations readily expose misunderstandings. Incorrect movement direction encodings and misplaced arrows directly reveal misinterpretations of the chain coding process. This demonstrates that visual explanations are not merely confirmations of errors; they are powerful diagnostic tools that uncover the root causes, leading to more effective analysis of AI-generated outputs.

Figure 4: Visualizations expose reasoning errors more clearly than text, facilitating easier diagnosis.

Dataset Details: TheoremExplainBench (TEB)

TheoremExplainBench (TEB) is a curated dataset designed to evaluate the capability of AI systems to generate multimodal explanations of theorems. The dataset comprises 240 theorems sourced from Computer Science, Chemistry, Mathematics, and Physics, providing a diverse assessment of reasoning and visualization skills.

Each theorem includes its name and a contextual description, sourced from OpenStax and LibreTexts. Theorems are categorized into Easy, Medium, and Hard difficulty levels, with 80 entries per category. TEB encompasses 68 distinct subfields, enabling focused analysis of AI performance across specific STEM domains.

Figure 5: Subfields of TheoremExplainBench under Computer Science, Chemistry, Mathematics, and Physics.

Citation

Please kindly cite our paper if you use our code, data, models or results:

@misc{ku2025theoremexplainagentmultimodalexplanationsllm,
                  title={TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding}, 
                  author={Max Ku and Thomas Chong and Jonathan Leung and Krish Shah and Alvin Yu and Wenhu Chen},
                  year={2025},
                  eprint={2502.19400},
                  archivePrefix={arXiv},
                  primaryClass={cs.AI},
                  url={https://arxiv.org/abs/2502.19400}, 
            }