Autoplay
Autocomplete
Previous Lesson
Complete and Continue
Applied Prompt Engineering: Prompt Testing & Model Benchmarks
Prompt Testing and Model Benchmarks
Introduction to Prompt Testing (8:32)
The Importance of Prompt Testing (4:32)
Thinking Like LLMs - Schrodinger's Prompt (19:17)
Building a Prompt Test (16:49)
The Golden Answer (7:00)
Model Benchmarks (11:29)
Deep Dive: MMLU Benchmark (14:19)
Prompt Tests vs. Model Benchmarks? (4:31)
The Latest: MMLU Pro (18:10)
Exercise: Create Your Own Mini-Benchmark (Totem Benchmark)!
Evaluating Results - Human Judge (7:03)
Evaluating Results - Code Judge (11:23)
Evaluating Results - AI Judge (22:06)
Deep Dive: LLMs as a Judge - Biases (22:11)
Deep Dive: LLMs as a Judge - Prompts (13:01)
LLM Judge Leaderboard
Thinking Like LLMs - Ex Post Facto Reasoning (11:46)
Applied Prompt Engineering - Developing Prompt Tests & Evaluations with PromptFoo
Here. We. Go! (6:01)
Introduction to PromptFoo (6:10)
Same As The Pros!
Our Prompt Testing Framework (4:45)
Warning!
Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Setup (23:09)
Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Results (10:20)
Housekeeping Matters
Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Setup (11:40)
Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Results (7:38)
Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Setup (25:36)
Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Results (8:21)
Optional Exercise: Practice Converting Another MMLU Dataset
Exercise: Explore Metrics (Assertions) in the PromptFoo Documentation
Exercise: Use Your Mini-Benchmark (Totem Benchmark)!
Our 4th Prompt Test (Adding Multiple Models) - The Setup (25:58)
Our 4th Prompt Test (Adding Multiple Models) - The Results (13:11)
Our 5th Prompt Test (Adding System Messages) - The Setup (24:31)
Our 5th Prompt Test (Adding System Messages) - The Results (12:22)
Do You Realize What Just Happened? (4:25)
Challenge - Become an AI Researcher!
Develop Your Own Prompting Technique
Introduction to Prompt Testing