Reinforcement Learning for Reasoning Models

Prompt Engineering Bootcamp (Working With AI & LLMs): Zero to Mastery

Section 0: Introduction

Learn To Work With AI & LLMs with Scott Kerr (1:00)
Course Introduction (6:19)
Course Resources + Handbook
Exercise: Meet Your Classmates and Instructor
Understanding Your Video Player
Set Your Learning Streak Goal

Section 1: Introduction to Prompt Engineering

What is Prompt Engineering? (6:24)
Why is Prompt Engineering Even a Thing? (8:31)
Breaking GPT (4:02)
Applied Prompt Engineering (4:24)
Applied Prompt Engineering with NASA (9:29)
Why is Prompt Engineering Important to You? (4:06)
The Latest: Am I Wrong?
Mandatory Homework: AlphaGo
Let's Have Some Fun (+ More Resources)

Section 2: Choose Your LLM

What I'm Using - Part 1 (1:21)
What I'm Using - Part 2 (OpenAI Playground) (4:02)
Optional: Setting Up Your Playground
Multi-Modality and Tools in LLMs (11:41)
Choose Your LLM!

ChatGPT Setup

Getting Started with ChatGPT (1:39)
The Basics of ChatGPT (4:08)
ChatGPT App (1:11)
Optional: ChatGPT Plus (1:55)

Claude Setup

Getting Started with Claude (4:27)

Open Source Option: Llama Setup

Getting Started with Llama

Section 3: Guided Project - Build Your First Game (Snake Game)

Project Introduction (2:13)
Important Point #1 - Fundamental Truths About LLMs (2:58)
Important Point #2 - Models are Non-Deterministic (4:03)
Setting Up Your Replit Account (4:44)
The First Try (4:41)
Building Our Snake Game - Part 1 (6:12)
Building Our Snake Game - Part 2 (5:49)
Exercise: Make It Prettier!
Unlimited Updates

Section 4: How LLMs Work

Introduction to LLMs (5:44)
Tokens (5:17)
Word Guessing Machines? (5:50)
Thinking Like LLMs - Roll a Dice (6:56)
Inside LLMs (22:13)
The Transformer Model (7:38)
GPT Meaning
Exercise: Visualize the LLM Architecture (3:56)
The Training Process (9:57)
Base Model vs. Assistant Model (7:06)
Thinking Like LLMs - The Reversal Curse (5:14)
Exercise: The Reversal Curse
Artificial General Intelligence (AGI) (5:28)
Exercise: Use ChatGPT to Read the Research (4:48)
The World of LLMs (6:05)
Implement a New Life System

Section 5: Challenge - Build Your Second Game (Tic Tac Toe with AI opponent)

Challenge Instructions

Section 6: Our Prompting Framework

Overview of Our Prompting Framework (3:30)
The Standard Prompt (4:33)
Prompt Library
Exercise: Get Hyped to Learn! (4:04)

Section 7: Prompting Fundamentals - The Setup

The Setup (1:01)
The System Message - Part 1 (6:07)
The System Message - Part 2 (5:43)
Exercise: Strengthen Your System Message (6:42)
Claude 3's System Message
Context - What it Is (5:17)
Context - The Context Window (9:21)
Context - Lost in the Middle (11:26)
The Latest: 1 Million Token Context Windows
Personas and Roles (8:42)
Personas - Tone, Style and Voice (11:49)
Exercise: Talk To Your Hero
Exercise: Write a Movie Scene
Custom Instructions (11:14)
The System Message for Custom Instructions
Thinking Like LLMs - Can GPT Keep a Secret? (4:45)
Exercise: Get ChatGPT to Spill the Tea - Part 1 (1:44)
Exercise: Get ChatGPT to Spill the Tea - Part 2 (1:42)
Exercise: Get ChatGPT to Spill the Tea - Part 3 (4:24)
Exercise: Test Techniques for Securing Secrets
Exercise: Stump Me!
Case Study: Moderna's Use of ChatGPT
Course Check-In

Section 8: Prompting Fundamentals - The Instructions

The User Message (2:09)
Be Clear and Specific (3:52)
Delimiters (5:15)
Exercise: Identify Delimiters (3:25)
X-Y Problem (6:10)
Exercise: Identify X
Solution: Identify X
In-Context Training (1:01)
Zero Shot Prompting (1:47)
One Shot and Few Shot Prompting (9:48)
Language Models Are Few-Shot Learners (3:48)
How Many Examples? (2:23)
Thinking Like LLMs - But Wait... (5:36)
Chain-of-Thought Prompting - Part 1 (5:24)
Chain-of-Thought Prompting - Part 2 (5:45)
Zero Shot CoT (9:29)
Exercise: Imposter Syndrome (2:55)

Section 9: Guided Project - Create Your Own Career Coach

Career Coach Ideation (7:32)
The Setup - Persona (6:02)
The Setup - Context and Commands (5:42)
The Instructions - Modes (12:20)
The Instructions - Chain of Thought (3:51)
The Instructions - Gamify (9:01)
Meet Your Career Coach (19:53)
Exercise: Customize Your Career Coach

Section 10: Prompting Fundamentals - The Output

The Output (1:35)
Length (12:33)
Formats (7:42)
Exercise: Output an Excel File (2:30)
Exercise: Make a Flowchart (4:01)

Section 11: The Dark Side of AI (Guardrails, Jailbreaks, Injections, Hallucinations + more)

Introduction to Guardrails and Jailbreaking (16:32)
Jailbreak! (The DAN Prompt) (7:25)
Exercise: Create Your Own Jailbreak Prompt
Many Shot Jailbreaking (18:09)
Prompt Injections - Part 1 (9:36)
Prompt Injections - Part 2 (17:42)
Thinking Like LLMs - Multi-Modal Injection (9:17)
Leaking - Part 1 (Prompt Leaking) (8:35)
Leaking - Part 2 (Data Leaking) (18:07)
Exposure (5:40)
Poisoning (3:18)
Toxicity (4:39)
Hallucinations (13:31)
Thinking Like LLMs - Big vs Small (18:58)
Challenge: Conduct Your Own Mechanistic Interpretability Research on Hallucinations (4:34)
Challenge Instructions
Leaderboard: Mechanistic Interpretability
The Model Card (11:05)
Model Cards Deep Dive (14:43)
Exercise: Explore the Latest Model Card and Learn Something New!

Section 12: LLM Hyperparameters & The OpenAI Playground

Introduction to Hyperparameters and the OpenAI Playground (3:15)
Note: Setting Up Your Playground
Playground UI Changes
Temperature (10:25)
Top P (4:08)
Frequency and Presence Penalties (4:38)
Stop Sequences (5:22)

Section 13: ChatGPT Deep Dive (Multi-Modality, Tools, Architecture and more)

Introduction (3:13)
Reminder: ChatGPT vs GPT
Custom Instructions (8:42)
Memory (10:38)
The Instruction Hierarchy (16:46)
The Latest: Reasoning Models - Use of Developer Message (1:53)
The Latest: Research To Prevent ChatGPT from Spilling the Tea (2:41)
Note re: Next Lesson
Replay: Multi-Modality and Tools in LLMs (11:41)
Multi-Modality Deep Dive (9:06)
Exercise: Examples of Each Multi-modal Combination
Thinking Like LLMs - Attack of the (Voice) Clones (18:22)
The Bigger Picture (9:27)
Tools & Function Calling - Part 1 (10:29)
Tools & Function Calling - Part 2 (13:42)
Function Calling Documentation
Wrap Up (3:07)

Section 14: Open Source Models

What Are Open Source LLMs and Why Are They Important? (6:50)
Chatbot Arena Leaderboard (7:52)
Battle of the Intelligence Tests (14:38)
Exercise: Test The Models With Your Specific Use-Case
Exercise: Create Your Own Test Prompt (Totem) (1:35)
My Totem Prompt
The Latest - The Best Open Source Model
Introduction to LMStudio (2:37)
Setting Up Your Own Model - Part 1 (2:51)
Setting Up Your Own Model - Part 2 (9:38)
Exercise: Explore the Guardrails

Section 15: Advanced Prompting Techniques

Introduction (1:06)
Auto-Priming (10:50)
Chain of Density Prompting (18:16)
Prompt Variables (7:13)
Prompt Chaining (8:38)
Prompt Chaining - Programmatic Visualization (17:40)
Exercise: Prompt Chaining - Customer Support (22:33)
Thinking Like LLMs - Dark Magic (9:57)
Exercise: Use Your Magic Powers
XML Tags - There's More Than Meets The Eye! (9:17)
Emotional Stimuli - Part 1 (18:36)
Emotional Stimuli - Part 2 (4:42)
Emotional Stimuli - Part 3 (But Why?) (7:17)
Self-Consistency (13:44)
Exercise: Implement the Self-Consistency Technique
ReAct Prompting (21:34)
ReAct + CoT-SC (16:47)
Applied Prompt Engineering with CRISPR - Part 1 (13:40)
Applied Prompt Engineering with CRISPR - Part 2 (14:52)
Tree of Thoughts - Part 1 (14:56)
Tree of Thoughts - Part 2 (4:56)
Tree of Thoughts - Part 3 (ToT via Code) (11:33)
Tree of Thoughts - Part 4 (ToT via Chaining) (22:11)
Tree of Thoughts - Part 5 ("Zero Shot" ToT) (12:26)

Section 16: Reasoning Models

Introduction (3:40)
A Note On The Next Two Lessons
Replay: Chain-of-Thought Prompting - Part 1 (5:24)
Replay: Chain-of-Thought Prompting - Part 2 (5:45)
Introduction to Reasoning Models (9:23)
First Contact with Reasoning (16:47)
Secrets and Lies! (12:14)
Setting Up Our Open Source Reasoning Model (5:51)
A Reasoning Model's Real Thoughts - Part 1 (5:15)
A Reasoning Model's Real Thoughts - Part 2 (8:40)
Exercise: Compare Reasoning Style of Different Models
Thinking Like LLMs - Breaking The Chains (12:15)
What Are Reasoning Models Good For? (The Generator-Verifier Gap) (13:32)
Exercise: Determine GVG (10:07)
Prompt Engineering for Reasoning Models (7:27)
Context Engineering (18:19)
Thinking Like LLMs: Cats Are...Confusing? - Part 1 (10:21)
Thinking Like LLMs: Cats Are...Confusing? - Part 2 (7:10)
Reinforcement Learning - The Problem (6:20)
Reinforcement Learning - How It Works (15:02)
Reinforcement Learning from Human Feedback (RLHF) (16:06)
Reinforcement Learning for Reasoning Models - Let's Verify Step-By-Step (6:36)
Reinforcement Learning for Reasoning Models - Process Reward Model (9:27)
PRM800K Introduction (7:41)
PRM800K Deep Dive (13:11)
Test-Time Compute (12:40)
Are Reasoning Models Lying To You? - Part 1 (11:07)
Are Reasoning Models Lying To You? - Part 2 (2:42)
Are Reasoning Models Lying To You? - Part 3 (7:53)
Are Reasoning Models Lying To You? - Part 4 (2:51)

Section 17: Challenge - Build Your Third Game (Flappy Bird)

Challenge Instructions
The Flappy Bird Hall of Fame

Section 18: Prompt Testing and Model Benchmarks

Introduction to Prompt Testing (8:32)
The Importance of Prompt Testing (4:32)
Thinking Like LLMs - Schrodinger's Prompt (19:17)
Building a Prompt Test (16:49)
The Golden Answer (7:00)
Model Benchmarks (11:29)
Deep Dive: MMLU Benchmark (14:19)
Prompt Tests vs. Model Benchmarks? (4:31)
The Latest: MMLU Pro (18:10)
Exercise: Create Your Own Mini-Benchmark (Totem Benchmark)!
Evaluating Results - Human Judge (7:03)
Evaluating Results - Code Judge (11:23)
Evaluating Results - AI Judge (22:06)
Deep Dive: LLMs as a Judge - Biases (22:11)
Deep Dive: LLMs as a Judge - Prompts (13:01)
LLM Judge Leaderboard
Thinking Like LLMs - Ex Post Facto Reasoning (11:46)

Section 19: Applied Prompt Engineering - Developing Prompt Tests & Evaluations with PromptFoo

Here. We. Go! (6:01)
Introduction to PromptFoo (6:10)
Same As The Pros!
Our Prompt Testing Framework (4:45)
Warning!
Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Setup (23:09)
Our 1st Prompt Test (Adding Prompts, Metrics + Human Judge) - The Results (10:20)
Housekeeping Matters
Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Setup (11:40)
Our 2nd Prompt Test (Adding New Prompts + Code Judge) - The Results (7:38)
Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Setup (25:36)
Our 3rd Prompt Test (Adding 152 Test Cases + AI Judge) - The Results (8:21)
Optional Exercise: Practice Converting Another MMLU Dataset
Exercise: Explore Metrics (Assertions) in the PromptFoo Documentation
Exercise: Use Your Mini-Benchmark (Totem Benchmark)!
Our 4th Prompt Test (Adding Multiple Models) - The Setup (25:58)
Our 4th Prompt Test (Adding Multiple Models) - The Results (13:11)
Our 5th Prompt Test (Adding System Messages) - The Setup (24:31)
Our 5th Prompt Test (Adding System Messages) - The Results (12:22)
Do You Realize What Just Happened? (4:25)

Section 21: AI Research and the Quest for Artificial General Intelligence

Introduction - Through The Looking Glass... (8:40)
Applied Prompt Engineering with The Turing Test (7:33)
Homework: Ex Machina
Mechanistic Interpretability - Part 1 (13:24)
Mechanistic Interpretability - Part 2 (20:59)
Scaling Laws - Model Size (12:55)
Scaling Laws - Dataset Size (8:04)
Scaling Laws - Training Compute (11:49)
Exercise: Current AI Trends
Homework: Situational Awareness - The Decade Ahead

Where To Go From Here?

What's Next?
Thank You! (1:17)
Review This Course!
Become An Alumni
Learning Guideline
ZTM Events Every Month
LinkedIn Endorsements

Optional Appendix: Guided Project - Prompting with Autonomous Agents (AutoGPT)

Project Introduction
A Quick Note
Setup Instructions
Setup Demo: Getting Your API Key (2:23)
Setup Demo: Downloading AutoGPT (1:03)
Setup Demo: Installing Docker (3:26)
Launching Your Autonomous Agent - Part 1 (3:25)
Launching Your Autonomous Agent - Part 2 (1:21)
It's Aliiiiiive! Running Your Autonomous Agent (4:51)
Task 1: Hello World - Your First Website (16:17)
Task 2: Python Program - Palindrome Checker (8:40)
Task 3: Initialize Your Own Task

Reinforcement Learning for Reasoning Models - Let's Verify Step-By-Step

This lecture is available exclusively for ZTM Academy members.

If you're already a member, you'll need to login.

Join ZTM To Unlock All Lectures