Example Curriculum
Prompt Testing and Model Benchmarks
Available in
days
days
after you enroll
- Introduction to Prompt Testing (8:32)
- The Importance of Prompt Testing (4:32)
- Thinking Like LLMs - Schrodinger's Prompt (19:17)
- Building a Prompt Test (16:49)
- The Golden Answer (7:00)
- Model Benchmarks (11:29)
- Deep Dive: MMLU Benchmark (14:19)
- Prompt Tests vs. Model Benchmarks? (4:31)
- The Latest: MMLU Pro (18:10)
- Exercise: Create Your Own Mini-Benchmark (Totem Benchmark)!
- Evaluating Results - Human Judge (7:03)
- Evaluating Results - Code Judge (11:23)
- Evaluating Results - AI Judge (22:06)
- Deep Dive: LLMs as a Judge - Biases (22:11)
- Deep Dive: LLMs as a Judge - Prompts (13:01)
- LLM Judge Leaderboard
- Thinking Like LLMs - Ex Post Facto Reasoning (11:46)
Applied Prompt Engineering - Developing Prompt Tests & Evaluations with PromptFoo
Available in
days
days
after you enroll
- Here. We. Go! (6:01)
- Introduction to PromptFoo (6:10)
- Same As The Pros!
- Our Prompt Testing Framework (4:45)
- Warning!
- Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Setup (23:09)
- Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Results (10:20)
- Housekeeping Matters
- Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Setup (11:40)
- Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Results (7:38)
- Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Setup (25:36)
- Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Results (8:21)
- Optional Exercise: Practice Converting Another MMLU Dataset
- Exercise: Explore Metrics (Assertions) in the PromptFoo Documentation
- Exercise: Use Your Mini-Benchmark (Totem Benchmark)!
- Our 4th Prompt Test (Adding Multiple Models) - The Setup (25:58)
- Our 4th Prompt Test (Adding Multiple Models) - The Results (13:11)
- Our 5th Prompt Test (Adding System Messages) - The Setup (24:31)
- Our 5th Prompt Test (Adding System Messages) - The Results (12:22)
- Do You Realize What Just Happened? (4:25)
Challenge - Become an AI Researcher!
Available in
days
days
after you enroll