I built a GH Action that uses AI to manually QA your PR using Magnitude/Claude

3 months ago 3

🤖 Automatically generate and run end-to-end tests for your Pull Requests using Claude API and Magnitude testing framework.

🔍 Smart Analysis: Analyzes PR changes and repository context
🧪 Auto-Generated Tests: Creates comprehensive E2E tests using Claude AI
⚡ Instant Execution: Runs tests immediately using Magnitude framework
💬 PR Integration: Comments results directly on your Pull Request
🎯 Zero Configuration: Works out of the box with sensible defaults

1. Add the workflow to your repository

Create .github/workflows/pr-tests.yml:

name: Auto-Generate PR Tests on: pull_request: types: [opened, synchronize] jobs: generate-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: ka-brian/pr-test-generator@v1 with: claude-api-key: ${{ secrets.CLAUDE_API_KEY }}

2. Configure your secrets

In your repository settings, add:

CLAUDE_API_KEY: Your Claude API key from Anthropic Console

Optional secrets for authentication:

TEST_USER_EMAIL: Email for test user authentication (if preview URLs require login)
TEST_USER_PASSWORD: Password for test user authentication (if preview URLs require login)

The action will automatically:

Analyze your PR changes
Generate relevant tests
Execute the tests
Comment the results on your PR

The action will add a comment to your PR like this:

## 🎉 Generated Tests PASSED *Auto-generated tests for PR #123 • 2025-07-18T10:30:00.000Z* ### Test Results: ✅ should navigate to updated dashboard ✅ should handle new form validation ✅ should display error messages correctly ### Test File: - **Location**: `.github/generated-tests/pr-123-tests.js` - **Status**: ✅ Tests executed successfully

Input Description Required Default

claude-api-key	Claude API key from Anthropic	✅	-
github-token	GitHub token for API access	❌	${{ github.token }}
test-examples	Custom test examples to guide Claude	❌	Built-in examples
output-dir	Directory for generated test files	❌	.github/generated-tests
timeout	Test execution timeout (seconds)	❌	120
comment-on-pr	Whether to comment results on PR	❌	true
wait-for-preview	Wait for preview URLs to appear in PR comments (seconds)	❌	60
base-url	Base URL to use for tests (overrides preview URL detection)	❌	-
test-user-email	Email for test user authentication (if preview requires login)	❌	-
test-user-password	Password for test user authentication (if preview requires login)	❌	-

Output Description

test-results	JSON string with test execution results
test-file-path	Path to generated test file
tests-passed	Boolean indicating if tests passed

- uses: yourusername/pr-test-generator@v1 with: claude-api-key: ${{ secrets.CLAUDE_API_KEY }} # Custom test examples for your project test-examples: | const { startBrowserAgent } = require('magnitude-core'); require('dotenv').config(); async function runTests() { const agent = await startBrowserAgent({ url: process.env.PREVIEW_URL || 'http://localhost:3000', narrate: true, browser: { launchOptions: { headless: true }, contextOptions: { viewport: { width: 1280, height: 720 } } }); try { console.log('Test: Loading dashboard'); await agent.act('Navigate to the dashboard page'); const heading = await agent.extract('Get the main dashboard heading text'); console.log('Dashboard heading:', heading); console.log('Test: User interactions'); await agent.act('Navigate to the homepage'); await agent.act('Click on the menu button'); await agent.act('Wait for the menu to open'); console.log('All tests completed successfully'); } finally { await agent.stop(); } } runTests().catch(error => { console.error('Test suite failed:', error); process.exit(1); }); # Save tests to custom location output-dir: "./e2e/generated" # Increase timeout for complex tests timeout: 300 # Disable PR commenting (use outputs instead) comment-on-pr: "false" # Wait longer for preview URLs to appear wait-for-preview: 120 # Override preview URL detection with specific base URL base-url: "https://my-staging-env.example.com" # Authentication for preview environments that require login test-user-email: ${{ secrets.TEST_USER_EMAIL }} test-user-password: ${{ secrets.TEST_USER_PASSWORD }}

Go to Anthropic Console
Create an API key
Add it to your repository secrets as CLAUDE_API_KEY

The action needs access to:

Read repository content and PR details
Write comments on Pull Requests
Execute tests in the GitHub Actions environment

This is handled automatically by the default GITHUB_TOKEN.

Preview URLs and Authentication

The action can automatically detect preview URLs from PR comments (Vercel, Netlify, Railway, etc.) and use them for testing. If your preview environments require authentication, you can configure test credentials:

Automatic Detection: The action scans PR comments for preview URLs
Manual Override: Use base-url input to specify a custom URL
Authentication: Set test-user-email and test-user-password for login-protected previews
Environment Variables: Tests can access credentials via process.env.TEST_USER_EMAIL and process.env.TEST_USER_PASSWORD

Supported Preview Platforms

Vercel (*.vercel.app)
Netlify (*.netlify.app)
Railway (*.railway.app)
Custom preview URLs matching pattern https://preview-*

Context Analysis: Fetches PR details, changed files, and repository structure
Test Generation: Sends context to Claude API with your test examples
Test Execution: Runs generated tests using Magnitude in a headless browser
Results Reporting: Comments detailed results on the PR

# Only run on PRs from the same repository (not forks) if: github.event.pull_request.head.repo.full_name == github.repository # Optional: only run on specific file changes on: pull_request: paths: - "src/**" - "components/**" - "pages/**"

# Run only on specific PR events on: pull_request: types: [opened, synchronize] # Skip 'ready_for_review', etc. # Set appropriate timeout with: timeout: 120 # Adjust based on your app's complexity

# Provide project-specific test patterns test-examples: | // Your actual test patterns const { startBrowserAgent } = require('magnitude-core'); require('dotenv').config(); async function runTests() { const agent = await startBrowserAgent({ url: process.env.PREVIEW_URL || 'http://localhost:3000', narrate: true, browser: { launchOptions: { headless: true }, contextOptions: { viewport: { width: 1280, height: 720 } } }); try { console.log('Test: User authentication'); await agent.act('Navigate to the login page'); await agent.act('Login with credentials', { data: { email: process.env.TEST_USER_EMAIL, password: process.env.TEST_USER_PASSWORD } }); await agent.act('Wait for redirect to dashboard'); console.log('Test: Access protected route'); await agent.act('Navigate to the admin section'); await agent.act('Wait for admin dashboard to load'); console.log('All tests completed successfully'); } finally { await agent.stop(); } } runTests().catch(error => { console.error('Test suite failed:', error); process.exit(1); });

Tests fail to execute