Stanford CS329Z | Engineering AI Agents

Instructors

Welcome!

The shift from monolithic language models to compound AI systems — systems with multiple interacting components including LLMs, retrievers, tools, and optimizers — represents a fundamental change in how AI applications are built for people. This course teaches students how to engineer agentic systems: the full spectrum from simple LLM pipelines to compound AI systems to autonomous agents. Students will learn to pick what types of problems to focus on, decompose problems, select appropriate components, collect and curate data, build evaluations, and reason about the design tradeoffs that arise when building these systems in practice.

Students first build core components (RAG, tool use, agent loops) from scratch, then learn how frameworks like DSPy abstract these patterns. Through three fully applied homework assignments and a quarter-long project, students gain hands-on experience building, optimizing, and evaluating agentic systems.

Class Schedule

Note: the schedule is tentative and subject to change. Classes meet Mondays and Wednesdays. Lecture materials will be linked here as they are released.

Week	Date	Lecture	Course Material
1	Wed Sep 23	Foundations & Landscape Introduction — What Are Agentic Systems? The spectrum from monolithic models to compound AI systems to agents; when compound systems win; the three engineering challenges (decomposition, data, evaluation); course logistics.	Readings: Zaharia et al. "The Shift from Models to Compound AI Systems." BAIR Blog (2024). Additional readings Ng. "Agentic Design Patterns Part 1: Four AI Agent Strategies…" The Batch (2024). Si et al. "Towards Execution-Grounded Automated AI Research." arXiv (2026).
2	Mon Sep 28	LLMs for Builders APIs & SDKs (litellm), structured I/O and constrained generation, decoding strategies and test-time compute, context engineering, model selection, and cost/latency tradeoffs.	Readings: Schluntz & Zhang. "Building Effective Agents." Anthropic (2024). Additional readings Rajasekaran et al. "Effective Context Engineering for AI Agents." Anthropic (2025).
2	Wed Sep 30	Building Blocks Retrieval-Augmented Generation (RAG) Grounding and hallucination, embeddings and vector stores, chunking strategies, hybrid search, cross-encoders and late interaction (ColBERT). Hands-on: build a RAG pipeline from scratch.	Readings: Lewis et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS (2020). Additional readings Khattab & Zaharia. "ColBERT: Efficient and Effective Passage Search via Late Interaction over BERT." SIGIR (2020).
3	Mon Oct 5	Tool Use & Function Calling The REPL, function-calling APIs, the Model Context Protocol (MCP), designing good tools, code-execution sandboxes, error handling and retries. Hands-on: build a tool-using system from scratch.	Readings: Model Context Protocol Specification. The Linux Foundation (2025).
3	Wed Oct 7	Frameworks & Agent Design Frameworks & Orchestration DSPy (signatures, modules, optimizers), LangChain/LangGraph, LlamaIndex; what frameworks abstract vs. what you built from scratch; choosing the right level of abstraction.	Readings: Khattab et al. "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines." ICLR (2024).
4	Mon Oct 12	Agent Design Patterns & Scaffolds The workflows-vs-agents taxonomy, five composable workflow patterns, agent patterns (ReAct, plan-and-execute, reflection), and scaffolds as design decisions.	Readings: Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR (2023).
4	Wed Oct 14	Memory & Multi-Agent Systems Agent Memory Architectures Short- vs. long-term memory, memory as tool-based actions, the file system as externalized memory, structured memory paradigms, and cross-agent memory.	Readings: Packer et al. "MemGPT: Towards LLMs as Operating Systems." arXiv (2023). Additional readings Letta. "Agent Memory: How to Build Agents that Learn and Remember." Letta Blog (2025). Chhikara et al. "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory." arXiv (2025). Park et al. "Generative Agents: Interactive Simulacra of Human Behavior." UIST (2023).
5	Mon Oct 19	Multi-Agent Systems Single vs. multi-agent architectures, orchestration patterns, handoffs and state transfer, delegation and collaboration patterns, and the challenges of coordination and error propagation.	Readings: Wu et al. "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." COLM (2024). Additional readings Cemri et al. "Why Do Multi-Agent LLM Systems Fail?" arXiv (2025). Neubig. "Don't Sleep on Single-agent Systems." OpenHands Blog (2024). Liu et al. "A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration." COLM (2024).
5	Wed Oct 21	Optimization Optimization The landscape from prompts to fine-tuning; prompt optimization (GEPA, MIPROv2, OPRO, TextGrad); test-time compute scaling; LoRA/QLoRA; distillation; RLHF/DPO at a high level; and when to optimize prompts vs. weights vs. inference compute.	Readings: Snell et al. "Scaling LLM Test-Time Compute Optimally…" ICLR (2025). Agrawal et al. "GEPA: Reflective Prompt Evolution Can Outperform RL." arXiv (2026). Additional readings Soylu et al. "Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together." EMNLP (2024). Opsahl-Ong et al. "Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs." arXiv (2024).
6	Mon Oct 26	📺 Guest Lecture (TBA)
6	Wed Oct 28	Data for Agentic Systems What Data Do Agents Need? Traces, demonstrations, and feedback; data for optimization vs. evaluation; data flywheels; synthetic data generation; collecting data from human-agent interaction.	Readings: Shankar. "Data Flywheels for LLM Applications." (2024). Additional readings Tan et al. "Large Language Models for Data Annotation and Synthesis: A Survey." EMNLP (2024).
7	Mon Nov 2	Data Selection & Quality Finding maximally informative data, filtering and selection strategies, tiny-but-targeted benchmarks, annotation practices, quality assessment, and building datasets from agent traces.	Readings: Yang et al. "SWE-smith: Scaling Data for Software Engineering Agents." NeurIPS D&B (2025). Shankar et al. "Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences." UIST (2024). Additional readings Zhou et al. "LIMA: Less Is More for Alignment." NeurIPS (2023).
7	Wed Nov 4	Evaluation for Agentic Systems Evaluation Fundamentals & Benchmark Design Why evals are hard, the 4-tuple framework (request, environment, stopping criteria, scorer), designing each component, properties of good benchmarks, realistic scaffolding, and reliability dimensions.	Readings: Press. "How to Build Good Language Modeling Benchmarks." (2024). Additional readings Polo et al. "tinyBenchmarks: Evaluating LLMs with Fewer Examples." ICML (2024).
8	Mon Nov 9	LLM-as-Judge & Evaluation Infrastructure The three grader types, designing judge prompts, known biases, pairwise vs. pointwise evaluation, non-determinism metrics (pass@k vs. pass^k), harness design, and Anthropic's 8-step roadmap.	Readings: Grace et al. "Demystifying Evals for AI Agents." Anthropic (2026). Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS (2023). Ryan et al. "AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators." ICLR (2026). Additional readings Zhu et al. "AutoLibra: Agent Metric Induction from Open-Ended Human Feedback." ICLR (2026).
8	Wed Nov 11	Safety Agent Safety & Guardrails Privacy risks of tool access, prompt injection (including indirect injection), red-teaming, sandboxing and permission models, output guardrails, liability considerations, responsible deployment, and human-in-the-loop patterns.	Readings: Shao et al. "PrivacyLens: Evaluating Privacy Norm Awareness of LMs in Action." NeurIPS D&B (2024). Zhang & Yang. "Searching for Privacy Risks in LLM Agents via Simulation." arXiv (2025). Li. "Agentic LLMs as Powerful Deanonymizers." arXiv (2026). Additional readings Wen et al. "Contextualized Privacy Defense for LLM Agents." arXiv (2026). OpenAI. "Understanding Prompt Injections: A Frontier Security Challenge." (2025). Anthropic. "Anthropic's Responsible Scaling Policy." (2023).
9	Mon Nov 16	📺 Guest Lecture (TBA)
9	Wed Nov 18	Coding Agents & Proactive Agents Coding & Software Agents How coding agents work end-to-end; SWE-agent, Claude Code, and OpenHands architectures; scaffolds as design decisions; SWE-bench and the 4-tuple framework in practice; the future of software development with agents.	Readings: Yang et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS (2024). Wang et al. "OpenHands: An Open Platform for AI Software Developers." ICLR (2025). Additional readings Anthropic. "Claude Code: Best Practices for Agentic Coding." (2025). Young. "Effective Harnesses for Long-Running Agents." Anthropic (2025). Jimenez et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR (2024).
10	Mon Nov 23	No Class — Thanksgiving Recess
10	Wed Nov 25	No Class — Thanksgiving Recess
11	Mon Nov 30	Proactive Agents From reactive to proactive; General User Models (GUM); Next Action Prediction; open-source proactive personal agents; privacy and trust implications; and when agents should initiate vs. wait (mixed initiative).	Readings: Shaikh et al. "Creating General User Models from Computer Use." UIST (2025). Additional readings Shaikh et al. "Learning Next Action Predictors from Human-Computer Interaction." arXiv (2026). Steinberger. "OpenClaw: Open-source proactive AI agent." (2025).
11	Wed Dec 2	Open Problems & Final Demos Frontiers & Open Problems Multimodal agents, web agents and computer use, science agents, long-running agent architectures, production and observability (tracing, monitoring, cost management), and open problems in reliability, scalability, and interpretability.	Additional readings Xie et al. "OSWorld: Benchmarking Multimodal Agents in Real Computer Environments." NeurIPS D&B (2024). Yao et al. "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents." NeurIPS (2022).
—	Finals Week (Dec 7–11)	Final Project Demo Day Held during the end-quarter examination period; exact time and location TBA.

Deadlines

All deadlines are tentative; exact dates and times are TBA.

Week	Deadline	Date	Time
3	HW1 Released	TBA	TBA
3	Project Proposal Due	TBA	TBA
5	HW1 Due	TBA	TBA
6	HW2 Released	TBA	TBA
6	Midway Report Due & Midpoint Demo	TBA	TBA
7	HW3 Released	TBA	TBA
8	HW2 Due	TBA	TBA
9	HW3 Due	TBA	TBA
Finals	Final Submission & Final System Demo	TBA	TBA

Coursework

Three fully applied homework assignments build on the components covered in lecture. Each homework is followed by a 10-minute oral exam where students explain their design decisions and tradeoffs and demonstrate understanding.

HW1: Build an Agentic System (Weeks 3–5). Given a repository of research papers, build an agent that answers science questions by retrieving relevant papers and reasoning over them. Part A builds the agent from scratch with litellm (RAG + tool use + an agent loop with a reasoning pattern such as ReAct); Part B rebuilds key components with DSPy and reflects on what the framework abstracts.
HW2: Data for Agents (Weeks 6–8). Given a staff-provided agent, collect and curate data to optimize its performance — data selection, quality filtering, finding maximally informative examples, synthetic data generation, and building optimization data (SFT or preference pairs). Deliverable: a curated dataset, a data card, and an analysis.
HW3: Evaluate an Agent (Weeks 7–9). Given a pre-built agent, design a comprehensive evaluation suite with code-based graders, at least one LLM-as-judge eval, benchmark tasks built with the 4-tuple framework (request, environment, stopping criteria, scorer), and error analysis.

Grading

Project (39%): proposal 8%, midway report 5%, midpoint demo 5%, final submission 8%, final system demo 15%
Oral check-ins (20%): 2 oral exams, 10% each
Homework (30%): 3 assignments, 10% each
Participation (5%): in-class discussion, project teamwork, and recitations

Course Project

Students work in groups on a quarter-long project on the theme "Making Life at Stanford Better with Agents." The goal is to build an agentic system that helps with some aspect of Stanford life. Example project ideas:

A syllabus reader that extracts deadlines and adds them to your calendar
A course-schedule optimizer for next quarter
A research-paper discovery and summarization agent
A campus-event aggregator and recommender

Milestones

Week 3: Project proposal (1-page)
Week 6: Midway report (1-page) + midpoint demo — a working prototype is expected
Week 10: Final submission (1-page) + final system demo (Demo Day)

Reports are brief (1–2 pages of writing, with an appendix for required structured content such as examples of your agent's failure modes).

Logistics

Course Info

Location: TBA
Timing: Mondays & Wednesdays; time TBA.
Units: 3–4
Contact: TBA

Office Hours

TBA — posted here and in the course forum before the first week of class.

Prerequisites

Any of CS224N, CS224U, CS224V, CS336, or equivalent background in natural language processing.

Honor Code

Collaboration, AI-tools, and academic-integrity policies: TBA.

Academic Accommodations

From Stanford's Office of Accessible Education: Students who may need an academic accommodation based on the impact of a disability must initiate the request with the Office of Accessible Education (OAE). Professional staff will evaluate the request with required documentation, recommend reasonable accommodations, and prepare an Accommodation Letter for faculty dated in the current quarter in which the request is being made. Students should contact the OAE as soon as possible since timely notice is needed to coordinate accommodations.

If you already have an Academic Accommodation Letter, we invite you to share your letter with us. Letters should be shared at the earliest possible opportunity so we may partner with you and the OAE to identify any barriers to access and inclusion.

Well-Being, Stress Management, & Mental Health

If you are experiencing personal, academic, or relationship problems and would like someone to talk to, reach out to Counseling and Psychological Services (CAPS) on campus. CAPS is the university's counseling center dedicated to student mental health and wellbeing. Phone assessment appointments can be made at CAPS by calling 650-723-3785, or by accessing the Vaden Patient portal. For more information, visit vaden.stanford.edu/caps-and-wellness.

CS 329Z: Engineering AI Agents

Stanford / Fall 2026