Show HN: Dagu – Self-contained, Powerful Alternative to Airflow v1.17.0 released

3 hours ago 1

v1.17.0-beta (2025-05-30) ​

Major Features ​

Improved Performance ​

  • Refactored execution history data for more performant history lookup
  • Optimized internal data structures for better scalability

Hierarchical DAG Execution ​

Execute nested DAGs with full parameter passing and output bubbling:

yaml

steps: - name: run_sub-dag run: sub-dag output: OUT params: "INPUT=${DATA_PATH}" - name: use output command: echo ${OUT.outputs.RESULT}

1
2
3
4
5
6
7
8

Multiple DAGs in Single File ​

Define multiple DAGs in one YAML file using --- separator:

yaml

name: main-workflow steps: - name: process run: sub-workflow # Defined below --- name: sub-workflow steps: - name: task command: echo "Hello from sub-workflow"

1
2
3
4
5
6
7
8
9
10
11

Parallel Execution with Parameters ​

Execute commands or sub-DAGs in parallel with different parameters for batch processing:

yaml

steps: - name: get files command: find /data -name "*.csv" output: FILES - name: process files run: process-file parallel: ${FILES} params: - FILE_NAME: ${ITEM}

1
2
3
4
5
6
7
8
9
10

Enhanced Web UI ​

  • Overall UI improvements with better user experience
  • Cleaner design and more intuitive navigation
  • Better performance for large DAG visualizations

New execution history page with:

  • Date-range filtering
  • Status filtering (success, failure, running, etc.)
  • Improved search performance
  • Better timeline visualization

Better Debugging ​

  • Precondition Results: Display actual results of precondition evaluations in the UI
  • Output Variables: Show output variable values in the UI for easier debugging
  • Separate Logs: stdout and stderr are now separated by default for clearer log analysis

Queue Management ​

Added enqueue functionality for both API and UI:

bash

# Queue a DAG for later execution dagu enqueue --run-id=custom-id my-dag.yaml # Dequeue dagu dequeue my-dag.yaml

1
2
3
4
5

Partial Success Status ​

New "partial success" status for workflows where some steps succeed and others fail, providing better visibility into complex workflow states.

API v2 ​

  • New /api/v2 endpoints with refactored schema
  • Better abstractions and cleaner interfaces
  • Improved error handling and response formats
  • See OpenAPI spec for details

Docker Improvements ​

Optimized Images ​

Thanks to @jerry-yuan:

  • Significantly reduced Docker image size
  • Split into three baseline images for different use cases
  • Better layer caching for faster builds

Container Enhancements ​

Thanks to @vnghia:

  • Allow specifying container name
  • Support for image platform selection
  • Better container management options

Enhanced Features ​

Advanced Repeat Policy ​

Thanks to @thefishhat:

  • Conditions for repeat execution
  • Expected output matching
  • Exit code-based repeats

yaml

steps: - name: wait for service command: check_service.sh repeatPolicy: condition: "${STATUS}" expected: "ready" intervalSec: 30 exitCode: [0, 1]

1
2
3
4
5
6
7
8

Bug Fixes & Improvements ​

  • Fixed history data migration issues
  • Improved error messages and logging
  • Better handling of edge cases in DAG execution
  • Performance improvements for large workflows
  • Various UI/UX enhancements: #925, #898, #895, #868, #903, #911, #913, #921, #923, #887, #922, #932, #962

Breaking Changes ​

DAG Type Field (v1.17.0-beta.13+) ​

Starting from v1.17.0-beta.13, DAGs now have a type field that controls step execution behavior:

  • type: chain (new default): Steps are automatically connected in sequence, even if no dependencies are specified
  • type: graph (previous behavior): Steps only depend on explicitly defined dependencies

To maintain the previous behavior, add type: graph to your DAG configuration:

yaml

type: graph steps: - name: task1 command: echo "runs in parallel" - name: task2 command: echo "runs in parallel"

1
2
3
4
5
6

Alternatively, you can explicitly set empty dependencies for parallel steps:

yaml

steps: - name: task1 command: echo "runs in parallel" depends: [] - name: task2 command: echo "runs in parallel" depends: []

1
2
3
4
5
6
7

Migration Required ​

History Data Migration: Due to internal improvements, history data from 1.16.x requires migration:

bash

# Migrate history data dagu migrate history

1
2

After successful migration, legacy history directories are moved to <DAGU_DATA_DIR>/history_migrated_<timestamp> for safekeeping.

Contributors ​

Huge thanks to our contributors for this release:

Installation ​

Try the beta version:

bash

# Docker docker run --rm -p 8080:8080 ghcr.io/dagu-org/dagu:latest dagu start-all # Or download specific version curl -L https://raw.githubusercontent.com/dagu-org/dagu/main/scripts/installer.sh | bash -s -- --version v1.17.0-beta

1
2
3
4
5

v1.16.0 (2025-01-09) ​

New Features ​

Enhanced Docker Image ​

  • Base image updated to ubuntu:24.04
  • Pre-installed common tools: sudo, git, curl, jq, python3, and more
  • Ready for production use with essential utilities

Dotenv File Support ​

Load environment variables from .env files:

yaml

dotenv: /path/to/.env # or multiple files dotenv: - .env - .env.production

1
2
3
4
5

🔗 JSON Reference Expansion ​

Access nested JSON values with path syntax:

yaml

steps: - name: sub workflow run: sub_workflow output: SUB_RESULT - name: use output command: echo "The result is ${SUB_RESULT.outputs.finalValue}"

1
2
3
4
5
6

If SUB_RESULT contains:

json

{ "outputs": { "finalValue": "success" } }

1
2
3
4
5

Then ${SUB_RESULT.outputs.finalValue} expands to success.

Advanced Preconditions ​

Regex Support: Use re: prefix for pattern matching:

yaml

steps: - name: some_step command: some_command preconditions: - condition: "`date '+%d'`" expected: "re:0[1-9]" # Run only on days 01-09

1
2
3
4
5
6

Command Preconditions: Test conditions with commands:

yaml

steps: - name: some_step command: some_command preconditions: - command: "test -f /tmp/some_file"

1
2
3
4
5

Enhanced Parameter Support ​

List Format: Define parameters as key-value pairs:

yaml

params: - PARAM1: value1 - PARAM2: value2

1
2
3

CLI Flexibility: Support both named and positional parameters:

bash

# Positional dagu start my_dag -- param1 param2 # Named dagu start my_dag -- PARAM1=value1 PARAM2=value2 # Mixed dagu start my_dag -- param1 param2 --param3 value3

1
2
3
4
5
6
7
8

Enhanced Continue On Conditions ​

Exit Code Matching:

yaml

steps: - name: some_step command: some_command continueOn: exitCode: [1, 2] # Continue if exit code is 1 or 2

1
2
3
4
5

Mark as Success:

yaml

steps: - name: some_step command: some_command continueOn: exitCode: 1 markSuccess: true # Mark successful even if failed

1
2
3
4
5
6

Output Matching:

yaml

steps: - name: some_step command: some_command continueOn: output: "WARNING" # Continue if output contains "WARNING" # With regex - name: another_step command: another_command continueOn: output: "re:^ERROR: [0-9]+" # Regex pattern matching

1
2
3
4
5
6
7
8
9
10
11

🐚 Shell Features ​

Piping Support:

yaml

steps: - name: pipe_example command: "cat file.txt | grep pattern | wc -l"

1
2
3

Custom Shell Selection:

yaml

steps: - name: bash_specific command: "echo ${BASH_VERSION}" shell: bash - name: python_shell command: "print('Hello from Python')" shell: python3

1
2
3
4
5
6
7
8

Sub-workflow Output ​

Parent workflows now receive structured output from sub-workflows:

json

{ "name": "some_subworkflow", "params": "PARAM1=param1 PARAM2=param2", "outputs": { "RESULT1": "Some output", "RESULT2": "Another output" } }

1
2
3
4
5
6
7
8

🔗 Simplified Dependencies ​

String format now supported:

yaml

steps: - name: first command: echo "First" - name: second command: echo "Second" depends: first # Simple string instead of array

1
2
3
4
5
6

Improvements ​

  • Environment Variable Expansion: Now supported in most DAG fields
  • UI Enhancements: Improved DAG visualization for better readability
  • Storage Optimization: Reduced state file sizes by removing redundant data

Bug Fixes ​

  • Fixed: DAGs with dots (.) in names can now be edited in the Web UI

Contributors ​

Thanks to our contributor for this release:

ContributionAuthor
Improved parameter handling for CLI - support for both named and positional parameters@kriyanshii

Previous Versions ​

For older versions, please refer to the GitHub releases page.

Version Support ​

  • Current: v1.16.x (latest features and bug fixes)
  • Previous: v1.15.x (bug fixes only)
  • Older: Best effort support

Migration Guides ​

Upgrading to v1.16.0 ​

Most changes are backward compatible. Key considerations:

  1. Docker Users: The new Ubuntu base image includes more tools but is slightly larger
  2. Parameter Format: Both old and new formats are supported
  3. State Files: Old state files are automatically compatible

Breaking Changes ​

None in v1.16.0

See Also ​

Read Entire Article
  1. Homepage
  2. Technology
  3. Show HN: Dagu – Self-contained, Powerful Alternative to Airflow v1.17.0 released

Related

Design Decisions in WarioWare DIY

Design Decisions in WarioWare DIY

16 minutes ago 0
Nobitex, Iran’s largest crypto exchange, suffered a major security breach

Nobitex, Iran’s largest crypto exchange, suffered a major se...

21 minutes ago 0
Atomic Discrepancy Could Be Hint of Fifth Force of Nature

Atomic Discrepancy Could Be Hint of Fifth Force of Nature

22 minutes ago 0
Hostinger Web Hosting