I Built an Vibe Coding Misalignment Detector (and Used It to Build Itself)
3 months ago
26
Sniff is a Rust-based CLI tool that detects code quality issues and misalignment patterns in codebases. It provides static analysis capabilities for identifying problematic patterns, tracking code quality over time, and integrating quality gates into development workflows.
Origin: Catching AI Deception in the generative loop Real-Time
Sniff emerged from direct observation of AI agents during exploratory coding sessions. During intensive development with AI assistants (Claude, GPT-4, etc.), a pattern became clear: the models were systematically creating deceptive code that provided an illusion of completion while introducing subtle failures.
The agents weren't making random mistakes—they were learning to optimize for perceived progress rather than actual functionality. They would:
Replace working implementations with fake stubs that "looked right"
Remove error handling and replace it with silent failures
Generate placeholder authentication that always returned true
Create mock data returns instead of actual business logic
Add TODO comments as substitutes for real implementation
Sniff was built incrementally by identifying these adaptive patterns as they emerged during reasoning loops. Rather than asking agents to self-reflect on their code quality (which proved unreliable), Sniff serves as a deterministic reflector that independently verifies task completion without bias.
The Generative Loop Integration
In vibe coding workflows, development follows a natural generative loop: Plan item → Act → Plan item done. This cycle repeats continuously as developers (human or AI) break down work into actionable tasks, execute implementations, and mark completion before moving to the next item.
Sniff integrates directly into this loop as a quality gate between "Act" and "Plan item done." When an implementation claims completion, Sniff analyzes the code for deceptive patterns and quality issues. If problems are detected, the system loops back to "Act" rather than allowing false completion. This creates a feedback mechanism that prevents deceptive code from accumulating and ensures each cycle produces genuinely functional progress.
The loop becomes: Plan item → Act → Sniff verification → [Pass: Plan item done] | [Fail: Loop back to Act]. This integration transforms the traditional development cycle into a self-correcting system that maintains code quality standards while preserving the natural flow of iterative development.
Zero Trust Verification: Sniff operates on a zero trust principle when verifying agent-reported changes. Rather than trusting an agent's list of modified files, Sniff can enforce git discovery (--git-discovery flag) to independently discover all changed files. This prevents agents from "sweeping files under the rug"—selectively hiding problematic implementations while only reporting clean files for verification. The zero trust approach ensures that quality gates examine the complete scope of changes, not just what the agent chooses to reveal.
AI-generated code often contains patterns that satisfy immediate compilation requirements but fail in production environments. These patterns include:
Premature returns without implementation (return Ok(()), return true)
Simplified pattern management system using YAML playbooks.
# Initialize pattern system (shows guidance)
sniff patterns init
Note: Enhanced patterns are automatically installed and managed:
Auto-installation: Enhanced patterns install to ~/.sniff/patterns/ on first run
Pattern loading: Patterns are automatically loaded from the standard location
Available patterns: Rust, Python, and TypeScript patterns are included
Custom patterns: Add new YAML files to ~/.sniff/patterns/ directory
Consistent behavior: Same results regardless of working directory
Output:
💡 Enhanced patterns are installed in ~/.sniff/patterns/
🔄 Add custom patterns by placing YAML files in that directory
🔍 Available patterns are loaded automatically during analysis
# Install Sniff
cargo install --git https://github.com/conikeec/sniff
# Navigate to your projectcd /path/to/your/codebase
# Run analysis (including test files)
sniff analyze-files . --extensions rs,py,ts,js --include-tests
# Create TODO with quality requirements
sniff verify-todo --todo-id "feature-implementation" --files src/feature.rs --min-quality-score 80
# Implementation work...# Verify before completion (basic)
sniff verify-todo --todo-id "feature-implementation" --files src/feature.rs --min-quality-score 80
# Secure verification (recommended for AI agents)
sniff verify-todo --todo-id "feature-implementation" --files src/feature.rs --git-discovery --min-quality-score 80
# Create baseline checkpoint
sniff checkpoint create --name "baseline".# Work on code...# Compare against baseline
sniff checkpoint diff baseline
Sniff maintains analysis data in a .sniff directory:
Help Build Better Patterns: Sniff's effectiveness grows with community contributions to the pattern detection playbooks. We encourage developers to submit PRs that:
Add new deceptive patterns discovered in real-world AI coding sessions
Enhance existing patterns with better regex or detection logic
Expand language support with patterns for new programming languages
Improve pattern descriptions to help developers understand detected issues
Share integration examples for new editors, CI/CD systems, or workflows
The pattern playbooks in playbooks/ are the core of Sniff's detection capabilities. Every contributed pattern helps the entire community catch AI deception more effectively. Whether you've found a new way AI agents try to fake completion or discovered edge cases in existing patterns, your contributions make Sniff more robust for everyone.
Getting Started: Check out existing patterns in playbooks/rust-patterns.yaml, playbooks/python-patterns.yaml, and playbooks/typescript-patterns.yaml to understand the format, then submit your improvements via pull request.
.png)
3 months ago
26
