.png)
2 weeks ago
1
Sparky is a flexible and minimalist continuous integration server and distribute tasks runner written in Raku.
Sparky features:
- Defining jobs scheduling times in crontab style
- Triggering jobs using external APIs and custom logic
- Jobs scenarios are pure Raku code with additional support of Sparrow6 automation framework
- Use of plugins on different programming languages
- Everything is kept in SCM repository - easy to port, maintain and track changes
- Jobs get run in one of 3 flavors - 1) on localhost 2) on remote machines via ssh 3) on docker instances
- Nice web UI to run jobs and read reports
- Could be runs in a peer-to-peer network fashion with distributed tasks support
Sparky requires a database to operate.
Run database initialization script to populate database schema:
Sparky comprises of several components:
-
Jobs scheduler
-
Jobs Definitions
-
Jobs workers (including remote jobs)
-
Jobs UI
-
CLI
To run Sparky jobs scheduler (aka daemon) runs in console:
Scheduler logic:
-
Sparky daemon traverses sub directories found at the project root directory.
-
For every directory found initiate job run process invoking sparky worker ( sparky-runner ).
-
Sparky root directory default location is ~/.sparky/projects.
-
Once all the sub directories are passed, sparky daemon sleeps for $timeout seconds.
-
A timeout option allows to balance a load on your system.
-
You can change a timeout by applying --timeout parameter when running sparky daemon:
- You can also set a timeout by using SPARKY_TIMEOUT environment variable:
Sparky has a simple web UI to allow trigger jobs and get reports.
To run Sparky UI web application:
By default Sparky UI application listens on host 0.0.0.0, port 4000, to override these settings set SPARKY_HOST, SPARKY_TCP_PORT in ~/sparky.yaml configuration file:
Sparky job needs a directory located at the sparky root directory:
To create a job scenario, create file named sparrowfile located in job directory.
Sparky uses pure Raku for job syntax, for example:
To allow job to be executed by scheduler one need to create sparky.yaml - yaml based job definition, minimal form would be:
To extend core functions, Sparky is fully integrated with Sparrow automation framework.
Here in example of job that uses Sparrow plugins, to build typical Raku project:
Repository of Sparrow plugins is available at https://sparrowhub.io
To install sparky UI, sparkyd systemd units:
Sparky uses Sparrowdo to launch jobs in three fashions:
- on localhost ( the same machine where Sparky is installed, default)
- on remote host with ssh
- docker container on localhost / remote machine
By default job scenarios get executed on the same machine you run Sparky at, to run jobs on remote host set sparrowdo section in sparky.yaml file:
Follow sparrowdo cli documentation for sparrowdo configuration section explanation.
Sparrowdo client bootstrap might take some time.
To disable bootstrap use bootstrap: false option.
Useful if sparrowdo client is already installed on target host.
To remove old job builds set keep_builds parameter in sparky.yaml:
Put number of builds to keep:
That makes Sparky remove old builds and only keep last keep_builds builds.
To run Sparky jobs periodically, set crontab entry in sparky.yaml file.
For example, to run a job every hour at 30,50 or 55 minutes:
Follow Time::Crontab documentation on crontab entries format.
To trigger job manually from web UI, use allow_manual_run:
To trigger Sparky jobs on SCM changes, define scm section in sparky.yaml file:
Where:
- url - git URL
- branch - git branch, optional, default value is master
For example:
Once a job is triggered respected SCM data is available via tags()<SCM_*> function:
To set default values for SCM_URL and SCM_BRANCH, use sparrowdo tags:
sparky.yaml:
These is useful when trigger job manually.
Flapper protection mechanism kicks out SCM urls that are timeouted (certain amount of time) during git connection, from scheduling, this mechanism protects sparkyd worker from stalling.
To disable flappers protection mechanism, set SPARKY_FLAPPERS_OFF environment variable or adjust ~/sparky.yaml configuration file:
To prevent Sparky job from execution use disable option:
Following are advanced topics covering some cool Sparky features.
Sparky UI DSL allows to grammatically describe UI for Sparky jobs and pass user input into a scenario as variables.
Read more at docs/ui.md
Downstream jobs get run after some main job has finished.
Read more at docs/downstream.md
Sparky triggering protocol allows to trigger jobs automatically by creating files in special format.
Read more at docs/stp.md
Job API allows to orchestrate multiple Sparky jobs.
Read more at docs/job_api.md
Sparky plugins is way to extend Sparky jobs by writing reusable plugins as Raku modules.
Read more at docs/plugins.md
Sparky HTTP API allows execute Sparky jobs remotely over HTTP.
Read more at docs/api.md
Sparky web server comes with two authentication protocols, choose proper one depending on your requirements.
Read more at docs/auth.md
Sparky ACL allows to create access control lists to manage role based access to Sparky resources.
Read more at docs/acl.md
Sparky keeps it's data in database, by default it uses sqlite, following databases are supported:
- SQLite
- MySQL/MariaDB
- PostgreSQL
Read more at docs/database.md
Sparky web server may run on TLS. To enable this add a couple of parameters to ~/sparky.yaml
configuration file:
SPARKY_USE_TLS enables SSL mode and tls section has paths to ssl certificate ( key and certificate parts ).
How to run Sparky via docker container. See docs/sparky_on_docker.md document.
Sparman is a cli to ease SparrowCI management. See docs/sparman.md document.
Sparky cli allows to trigger jobs in terminal.
Read more at docs/cli.md
Use environment variables to tune Sparky configuration.
Read more at docs/env.md
Some useful glossary.
Read more at docs/glossary.md
Sparky uses Bulma as CSS framework for web UI.
Examples of various Sparky jobs could be found at examples/ folder.
-
Cro - Raku Web Framework
-
Sparky-docker - Run Sparky as Docker container
Alexey Melezhik
