Show HN: PromptDrifter – Catch LLM prompt drift before it breaks prod

🏃 Quick-Start - ❓ Why PromptDrifter? - 🎬 Demo - 📚 Docs
🐛 Bug Report - ✨ Feature Request

To start using PromptDrifter:

1. Initialize a sample configuration:

This will create a promptdrifter.yaml file in your current directory.

2. Edit promptdrifter.yaml to define your prompts and expected outcomes.

3. Run your tests:

   promptdrifter run promptdrifter.yaml

The landscape of Large Language Models (LLMs) is one of rapid evolution. While exciting, this constant change introduces a critical challenge for applications relying on them: prompt drift.

Over time, updates to LLM versions, or even subtle shifts in their training data or internal architecture, can cause their responses to identical prompts to change. These changes can range from minor formatting differences to significant alterations in content or structure, potentially breaking downstream processes causing issues with the integrity of your application.

---

🚨 Unexpected Failures Applications or CI/CD pipelines may break silently or with cryptic errors when LLM outputs deviate from expected formats or content. 📉 Degraded User Experience Features relying on consistent LLM responses can malfunction, leading to user frustration. ⏱️ Increased Maintenance Engineers spend valuable time diagnosing issues, tracing them back to changed LLM behavior rather addressing features and bugs in code. 🚧 Blocked Deployments Uncertainty about LLM stability can slow down development cycles and deployment frequency.

---

🛡️ Automated Guardrails A simple, command-line driven tool to integrate LLM response validation directly into your development and CI/CD workflows. 🔍 Early Drift Detection By comparing LLM outputs against version-controlled expected responses or predefined patterns (like regex), **PromptDrifter** catches deviations as soon as they occur. ⚙️ Consistent and Reliable Applications Ensures that your LLM-powered features behave predictably by failing builds when significant response changes are detected, *before* they impact users or production systems. 🔌 Model Agnostic Design Through a flexible adapter system, PromptDrifter can interact with various LLM providers and models (e.g., OpenAI, Ollama, and more to come). 📝 Declarative Test Suites Define your prompt tests in easy-to-understand YAML files, making them simple to create, manage, and version alongside your codebase. 😌 Developer Peace of Mind Build with greater confidence, knowing you have a safety net that monitors the stability of your critical prompt interactions.

PromptDrifter is designed to be extensible to various Large Language Models through its adapter system. Here's a current list of supported and planned adapters:

Provider / Model Family Adapter Status Details / Model Examples Linked Issue

GPT (OpenAI)	✅ Available	gpt-3.5-turbo, gpt-4, gpt-4o, etc.	N/A
Ollama	✅ Available	llama3, mistral, gemma, etc.	N/A
Claude (Anthropic)	✅ Available	claude-3-7-sonnet, claude-3-5-sonnet, claude-3-opus	N/A
Gemini (Google)	✅ Available	gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash-thinking	N/A
Qwen (Alibaba)	✅ Available	qwen3-30b-a3b, qwq-32b	N/A
Grok (xAI)	✅ Available	grok-3, grok-2, etc.	N/A
DeepSeek	✅ Available	deepseek-r1, deepseek-v3-0324, etc.	N/A
Mistral	✅ Available	mistral-small-24b-instruct-2501, mistral-small-3.1-24b-instruct-2503	N/A
Azure OpenAI	📋 To Do	gpt-3.5-turbo, gpt-4, gpt-4o, etc.	#10
Llama (Meta)	📋 To Do	llama-4-maverick, llama-4-scout, etc.	#11

If there's a model or provider you'd like to see supported, please open a feature request or consider contributing an adapter!

Name Config key Description Installation

Exact Match	expect_exact	Output should be an exact match	✅ Core
Regex	expect_regex	Output should match regex pattern	✅ Core
Substring	expect_substring	Output should contain the substring	✅ Core
Substring Case Insensitive	expect_substring_case_insensitive	Case insensitive substring match	✅ Core
Text Similarity	text_similarity	Semantic similarity using sentence transformers	pip install 'promptdrifter[similarity]'

Automate your prompt drift detection by integrating PromptDrifter directly into your GitHub workflows!

We provide a reusable GitHub Action that makes it easy to run your PromptDrifter tests on every push or pull request.

➡️ Find the PromptDrifter GitHub Action here: PromptDrifter on GitHub Marketplace

Add this to your workflow file (e.g., .github/workflows/prompt-drift.yml):

This action allows you to:

• Install a specific version of PromptDrifter or use the latest
• Specify your test files and configurations
• Support multiple LLM providers (OpenAI, Claude, Gemini, etc.)
• Secure API key handling via GitHub secrets

By using the action, you can ensure that any changes to your LLM's responses that violate your defined tests will automatically flag your CI builds, preventing unexpected issues from reaching production.

Our documentation is built with Docusaurus and deployed to GitHub Pages.

You can view the full documentation at: https://code-and-sorts.github.io/PromptDrifter/

The documentation includes:

• Getting Started guide
• Configuration options
• API reference
• Examples and tutorials

If you want to contribute to the documentation, see the docs/README.md file for instructions.

Follow the contributing guide.

Please make sure you read the Code of Conduct guide.

• 0.0.x