Roast: A convention-oriented framework for creating structured AI workflows

A convention-oriented framework for creating structured AI workflows, maintained by the Augmented Engineering team at Shopify.

Roast provides a structured, declarative approach to building AI workflows with:

• Convention over configuration: Define powerful workflows using simple YAML configuration files and prompts written in markdown (with ERB support)
• Built-in tools: Ready-to-use tools for file operations, search, and AI interactions
• Ruby integration: When prompts aren't enough, write custom steps in Ruby using a clean, extensible architecture
• Shared context: Each step shares its conversation transcript with its parent workflow by default
• Step customization: Steps can be fully configured with their own AI models and parameters.
• Session replay: Rerun previous sessions starting at a specified step to speed up development time
• Parallel execution: Run multiple steps concurrently to speed up workflow execution
• Function caching: Flexibly cache the results of tool function calls to speed up workflows
• Extensive instrumentation: Monitor and track workflow execution, AI calls, and tool usage (see instrumentation documentation)

Here's a simple workflow that analyzes test files:

Each step can have its own prompt file (e.g., analyze_coverage/prompt.md) and configuration. Steps can be run in parallel by nesting them in arrays:

Workflows can include steps that run bash commands (wrap in $()), use interpolation with {{}} syntax, and even simple inlined prompts as a natural language string.

1. Create a workflow YAML file defining your steps and tools
2. Create prompt files for each step (e.g., step_name/prompt.md)
3. Run the workflow:

In Roast, workflows maintain a single conversation with the AI model throughout execution. Each step represents one or more user-assistant interactions within this conversation, with optional tool calls. Steps naturally build upon each other through the shared context.

Roast supports several types of steps:

1. Standard step: References a directory containing at least a prompt.md and optional output.txt template. This is the most common type of step.

As an alternative to a directory, you can also implement a custom step as a Ruby class, optionally extending Roast::Workflow::BaseStep.

In the example given above, the script would live at workflow/analyze_code.rb and should contain a class named AnalyzeCode with an initializer that takes a workflow object as context, and a call method that will be invoked to run the step. The result of the call method will be stored in the workflow.output hash.

2. Parallel steps: Groups of steps executed concurrently

   steps: - - analyze_code_quality - check_test_coverage

3. Command execution step: Executes shell commands directly, just wrap in $(expr)

   steps: - $(command line expr) - rubocop: $(bundle exec rubocop -A)

   This will execute the command and store the result in the workflow output hash. Explicit key name is optional (rubocop in the second line of the example).

4. Raw prompt step: Simple text prompts for the model without tools

   steps: - Summarize the changes made to the codebase.

   This creates a simple prompt-response interaction without tool calls or looping. It's detected by the presence of spaces in the step name and is useful for summarization or simple questions at the end of a workflow.

Roast handles data flow between steps in three primary ways:

1. Conversation Context (Implicit): The LLM naturally remembers the entire conversation history, including all previous prompts and responses. In most cases, this is all you need for a step to reference and build upon previous results. This is the preferred approach for most prompt-oriented workflows.

2. Output Hash (Explicit): Each step's result is automatically stored in the workflow.output hash using the step name as the key. This programmatic access is mainly useful when:

   • You need to perform non-LLM transformations on data
   • You're writing custom output logic
   • You need to access specific values for presentation or logging

3. Interpolation (Dynamic): You can use {{expression}} syntax to inject values from the workflow context directly into step names, commands, or prompt text. For example:

   steps: - analyze_file - $(rubocop -A {{file}}) - Generate a summary for {{file}} - result_for_{{file}}: store_results

   Interpolation supports:

   • Simple variable access: {{file}}, {{resource.target}}
   • Access to step outputs: {{output['previous_step']}}
   • Any valid Ruby expression evaluated in the workflow context: {{File.basename(file)}}

For typical AI workflows, the continuous conversation history provides seamless data flow without requiring explicit access to the output hash. Steps can simply refer to previous information in their prompts, and the AI model will use its memory of the conversation to provide context-aware responses. For more dynamic requirements, the interpolation syntax provides a convenient way to inject context-specific values into steps.

• -o, --output FILE: Save results to a file instead of outputting to STDOUT
• -c, --concise: Use concise output templates (exposed as a boolean flag on workflow)
• -v, --verbose: Show output from all steps as they execute
• -r, --replay STEP_NAME: Resume a workflow from a specific step, optionally with a specific session timestamp

The session replay feature allows you to resume workflows from specific steps, saving time during development and debugging:

Sessions are automatically saved during workflow execution. Each step's state, including the conversation transcript and output, is persisted to a session directory. The session directories are organized by workflow name and file, with timestamps for each run.

This feature is particularly useful when:

• Debugging specific steps in a long workflow
• Iterating on prompts without rerunning the entire workflow
• Resuming after failures in long-running workflows

Sessions are stored in the .roast/sessions/ directory in your project. Note that there is no automatic cleanup of session data, so you might want to periodically delete old sessions yourself.

The target option is highly flexible and accepts several formats:

Single file path:

Directory path:

Glob patterns:

URL as target:

API configuration (Fetch API-style):

Shell command execution with $(...):

Git integration examples:

Roast also supports workflows that don't operate on a specific pre-defined set of target files:

API-driven workflows:

Data generation workflows:

These targetless workflows are ideal for:

• API integrations
• Content generation
• Report creation
• Interactive tools
• Scheduled automation tasks

You can set a default model for all steps in your workflow by specifying the model parameter at the top level:

Individual steps can override this setting with their own model parameter:

Roast supports both OpenAI and OpenRouter as API providers. By default, Roast uses OpenAI, but you can specify OpenRouter:

Benefits of using OpenRouter:

• Access to multiple model providers through a single API
• Support for models from Anthropic, Meta, Mistral, and more
• Consistent API interface across different model providers

When using OpenRouter, specify fully qualified model names including the provider prefix (e.g., anthropic/claude-3-opus-20240229).

Roast allows you to dynamically fetch API tokens using shell commands directly in your workflow configuration:

This makes it easy to use environment-specific tokens without hardcoding credentials, especially useful in development environments or CI/CD pipelines. Alternatively, Roast will fall back to OPENROUTER_API_KEY or OPENAI_API_KEY environment variables based on the specified provider.

Each step can have an output.txt file that uses ERB templating to format the final output. This allows you to customize how the AI's response is processed and displayed.

Example step_name/output.txt:

This is an example of where the workflow.output hash is useful - formatting output for display based on data from previous steps.

Available in templates:

• response: The AI's response for this step
• workflow: Access to the workflow object
• workflow.output: The shared hash containing results from all steps when you need programmatic access
• workflow.file: Current file being processed (or nil for targetless workflows)
• All workflow configuration options

For most workflows, you'll mainly use response to access the current step's results. The workflow.output hash becomes valuable when you need to reference specific data points from previous steps in your templates or for conditional display logic.

Roast provides extensive instrumentation capabilities using ActiveSupport::Notifications. You can monitor workflow execution, track AI model usage, measure performance, and integrate with external monitoring systems. Read the full instrumentation documentation.

Roast provides several built-in tools that you can use in your workflows:

Reads the contents of a file from the filesystem.

• The path can be absolute or relative to the current working directory
• Use offset and limit for large files to read specific sections (line numbers)
• Returns the file content as a string

Writes content to a file, creating the file if it doesn't exist or overwriting it if it does.

• Creates missing directories automatically
• Can restrict file operations to specific directories for security
• Returns a success message with the number of lines written

Applies a unified diff/patch to one or more files. Changes are applied atomically when possible.

• Accepts standard unified diff format from tools like git diff
• Supports multiple file changes in a single operation
• Handles file creation, deletion, and modification
• Performs atomic operations with rollback on failure
• Includes fuzzy matching to handle minor context differences
• This tool is especially useful for making targeted changes to multiple files at once

Searches file contents for a specific pattern using regular expressions.

• Uses regular expressions for powerful pattern matching
• Can filter by file types using the include parameter
• Can scope searches to specific directories with the path parameter
• Returns a list of files containing matches

Provides advanced file search capabilities beyond basic pattern matching.

• Combines pattern matching with contextual search
• Useful for finding specific code structures or patterns
• Returns matched lines with context

Executes shell commands and returns their output.

• Provides access to shell commands for more complex operations
• Can specify working directory and environment variables
• Captures and returns command output
• Useful for integrating with existing tools and scripts

Creates a specialized agent for complex coding tasks or long-running operations.

• Delegates complex tasks to a specialized coding agent
• Useful for tasks that require deep code understanding or multi-step changes
• Can work across multiple files and languages

You can create your own tools using the Raix function dispatch pattern. Custom tools should be placed in .roast/initializers/ (subdirectories are supported):

Then include your tool in the workflow:

The tool will be available to the AI model during workflow execution, and it can call analyze_commit with the appropriate parameters.

You can extend Roast with project-specific configuration by creating initializers in .roast/initializers/. These are automatically loaded when workflows run, allowing you to:

• Add custom instrumentation
• Configure monitoring and metrics
• Set up project-specific tools
• Customize workflow behavior

Example structure:

Or add to your Gemfile:

After checking out the repo, run bundle install to install dependencies. Then, run bundle exec rake to run the tests and linter. You can also run bin/console for an interactive prompt that will allow you to experiment.

The gem is available as open source under the terms of the MIT License.