Roast: Structured AI Workflows Made Easy

This past year, we started exploring how AI agents could boost developer productivity. In thinking about big developer problems such as flaky tests or lack of test coverage, we found immediate opportunities in grading and optimizing unit tests at scale, with minimal human intervention. Once we began implementing AI workflows, we quickly learned that AI agents need help staying on track, and work much better when you break down complicated prompts into discrete steps. Allowing AI to roam free around millions of lines of code just didn’t work very well. Non-determinism is the enemy of reliability.

To address this challenge, and give AI the structure it needs, the Augmented Engineering Developer Experience (DX) team at Shopify built and open-sourced a new tool called Roast.

What is Roast?

Roast is a convention-oriented workflow orchestration framework designed specifically for creating structured AI workflows that interleave non-deterministic AI behavior with normal non-AI code execution. It provides a declarative approach using YAML configuration and markdown prompts, giving AI agents the guardrails needed to solve developer productivity problems at scale.We extracted Roast from our internal AI tools once we realized its potential to help the broader developer community inside and outside of the company. And while Roast is itself implemented in Ruby, it’s a command line tool that can be used with any other programming languages.

What makes Roast most exciting is its ability to turn complex, multi-step agentic AI processes into reproducible, testable workflows. Whether you're analyzing code quality, generating documentation, or anything else that involves a series of AI and non-AI steps, Roast provides the structure to make the AI parts work reliably at scale. Roast workflows can be version-controlled, tested, and integrated into development workflows easily.

By open-sourcing Roast, we have invited the community to help shape the future of AI-assisted task execution. We believe that structured AI workflows will become as essential to modern development as CI/CD pipelines are today. As a result, a dozen Engineers have already contributed features and workflows to Roast on GitHub.

The name "Roast" comes from our initial use case: roasting your tests to find areas for improvement. Just like a good roast brings out hidden flavors, Roast helps uncover opportunities to enhance your code quality. I have also joked that Roast helps you set your money on fire as you find new ways to burn millions of tokens, but I digress.

Convention over Configuration

Roast follows Ruby on Rails' philosophy of convention over configuration. Simply create a workflow.yml file and corresponding prompt files, and you're ready to go:

Roast intelligently interprets different step formats based on their structure:

1. Directory-based Steps

The most common type - create a directory with the step name containing a prompt.md file:

The prompt file can use ERB templating to access workflow context:

2. Command Execution Steps

Wrap shell commands in $() to execute them and capture output:

3. Inline Prompt Steps

Any string with spaces becomes a direct prompt to the AI model associated with the step. Prefix an inline prompt with a ^ character to prompt Roast’s built-in Coding Agent tool (powered by Claude Code):

4. Custom Ruby Steps

For complex logic, Roast allows creation of Ruby step classes that inherit from BaseStep:

5. Parallel Steps

Use nested arrays to run some steps of your workflow concurrently:

Built-in Tools

Roast provides a comprehensive toolkit out of the box:

ReadFile: Reads file contents with line numbers
WriteFile: Writes content to files with security restrictions
UpdateFiles: Applies diffs/patches to multiple files
Grep: Searches file contents using regex patterns
SearchFile: Advanced file search with glob patterns
Cmd: Executes shell commands (with configurable restrictions)
Bash: Unix command execution with proper error handling
CodingAgent: Our most powerful tool—integrates Claude Code directly into workflows

The CodingAgent: Roast's Secret Weapon

The CodingAgent is what truly sets Roast apart. It's not just another tool—it's a full integration with Claude Code that brings agentic capabilities into structured workflows. This creates a powerful hybrid approach where you get the best of both worlds:

The CodingAgent excels at tasks that require iteration and adaptation:

Complex code modifications that need multiple attempts to get right
Bug fixing where the solution requires exploration and testing
Performance optimization through iterative improvements
Test generation that adapts based on coverage feedback

What makes this revolutionary is how it combines Roast's structured approach with Claude Code's adaptive problem-solving. You define the guardrails and objectives, but the agent has autonomy within those boundaries to iterate, test, and improve until it achieves the goal.

For example, in our workflow for adding Sorbet types (known as “Boba”), we use deterministic steps to clean up the code and run Sorbet's autocorrect, then hand off remaining issues to the CodingAgent. It will iteratively fix type errors, run tests, and ensure everything passes—something that would be nearly impossible with pure deterministic automation.

Shared Context & Smart Data Flow

Steps in a workflow share their conversation transcript, building upon each other's context. This allows for sophisticated workflows where later steps can automatically reference and build upon earlier discoveries without any additional configuration by the workflow author needed:

Advanced Control Flow

Roast supports sophisticated control structures that go beyond simple sequential execution:

Iteration over collections:

Conditional execution:

Case statements for multi-branch logic:

Session Replay & Development Experience

One of Roast's killer features is session replay. Every workflow execution is automatically saved, allowing you to resume from any step:

This dramatically speeds up workflow development by eliminating the need to rerun expensive AI operations over and over again.

Example: Grading a Ruby Unit Test

Let's walk through a real example that showcases Roast's power. Our test grading workflow acts like a senior engineer reviewing your tests:

When you run this workflow:

You get a comprehensive report:

Getting Started with Roast

Roast is distributed as a Ruby gem, so getting started is pretty straightforward:

Prerequisites:

Ruby 3.0+
An OpenAI API key (or OpenRouter for other models)
Optional: shadowenv and ripgrep for enhanced functionality

Built on Raix (Ruby AI eXtensions)

Roast leverages Raix, a library that provides a powerful abstraction layer for AI interactions. Raix enables Roast to work seamlessly with different AI providers and adds advanced features like retry logic, response caching, and structured output handling.

You can customize Raix behavior through initializers in your .roast/initializers directory. Here's an example that adds automatic retry logic and debugging:

This integration gives you fine-grained control over AI interactions, including:

Custom authentication schemes
Request retry strategies
Response logging and debugging
Provider-specific configurations
Token usage tracking

Real-World Impact at Shopify

Since deploying Roast internally, we've seen some fascinating use cases emerge:

Test Quality at Scale: Our engineers have analyzed thousands of test files, automatically identifying and fixing common antipatterns and significantly increasing test coverage across the board.

Automated Type Safety with Boba: We created a Roast workflow called "Boba" that automatically adds Sorbet type annotations to our test files. The workflow performs cleanup with sed, bumps tests to strict typing, runs Sorbet's autocorrect, and then feeds any remaining issues to our coding agent. For example, running boba fully typed a test file that previously had no type annotations, ensuring both tests and type checking pass.

Proactive Site Reliability Monitoring: Our SRE team uses Roast to periodically scan internal Slack channels for early indicators of emerging issues. The workflow analyzes conversation patterns, identifies potential problems before they escalate, and alerts the appropriate teams—turning reactive firefighting into proactive incident prevention.

Competitive Intelligence Aggregation: We built a Roast workflow that gathers and consolidates competitive information from multiple sources:

News and market analysis from third-party web sources
Migration patterns and brand switching data
Market trends (gathered via API from customer conversations captured in CRM tools) The workflow synthesizes this disparate data into actionable intelligence reports, saving hours of manual research.

"Chesterton's Fence" Code Research Tool: One of our most innovative workflows helps developers understand the historical context of code decisions. When encountering puzzling code, developers can run a Roast workflow that researches the commit history, analyzes related PRs, and explains why particular lines of code exist—preventing the removal of seemingly unnecessary but actually critical code.

The Future of Structured AI Workflows

When preparing this blog post, I found an amazing quote from Sam Schmidt that captures something truly innovative about Roast:

This is revolutionary thinking. Traditional development forces you to fully understand a problem before automating it. Roast flips this on its head—you can use AI as a placeholder to keep your workflow moving, then replace it with deterministic code once you understand the problem better. It's like having a junior developer who can handle the parts you haven't figured out yet, allowing you to prototype entire workflows before committing to specific implementations. This approach dramatically accelerates the development of complex automation.

We believe Roast represents the future of how developers will work with AI. Rather than ad-hoc prompts in chat interfaces, we see a world where:

AI workflows are first-class citizens in the development process, version-controlled and tested like any other code
Hybrid workflows seamlessly blend deterministic steps with AI-powered analysis
Workflow marketplaces emerge where teams share battle-tested workflows for common tasks
AI becomes predictable and reliable when it needs to be through structured execution rather than free-form generation

Our dream is to see Roast workflows become as ubiquitous as GitHub Actions or Jenkins pipelines—a standard part of every development team's toolkit.

Join Us

Roast is more than just a tool, it's a new paradigm for working with AI in software development. We invite you to:

Try Roast on your own codebase
Share your workflows with the community
Contribute to the framework's development
Join the conversation about the future of AI-assisted development

Special thanks to the entire Augmented Engineering team at Shopify who made this possible, with particular recognition to Paulo Arruda, Sam Schmidt, Josh Nichols, and the many internal users whose feedback shaped Roast into what it is today.

The age of structured AI workflows is here. Let's build it together.

Roast is available now at github.com/Shopify/roast. For questions and contributions, please see our contributing guidelines.