Background Tasks feature merged into Django

&LTnote>

Note that django.tasks and django-tasks are in fact different, albeit by a single character. Take care when reading.

&LT/note>

#What happened?

As of today, the first PR to implement DEP 14 ("Background workers") has been merged into the main branch of the Django code base, all 3094 lines of it. We missed the 5.2 release window last year, but we're here now. django.tasks and bring background workers to Django is, in my humble opinion, one of the largest and most important steps forward for Django since the introduction of ASGI in Django 3.0. django.tasks will be available for general use starting with the 6.0 release, due out later this year. The 6.0 code freeze is today (2025-09-17), which means it only just made it in. But, it's also not finalised yet - it could theoretically be taken out, but I have everything crossed that it won't.

No more do you need to define your background tasks with backend-specific APIs, or rewrite all your tasks as you scale:

If you've been following the developments of DEP 14, specifically those in django-tasks, you'll notice some glaring omissions from what was just merged. Not everything has been merged in to Django yet. The main interfaces, including backend handling and Task definitions are both included, along with the "Immediate" and "Dummy" backends - not to mention a lot of documentation. The more complex backends, such as the custom database backend and the RQ compatibility backend are left out, very much intentionally. The database backend will absolutely come in time - it just needs more features and finishing touches before it's stable enough for the likes of django itself. For now however, you can install it from django-tasks.

#Thanks

Over the past 585 days, I've spent a lot of my spare time working on DEP 14. Writing the DEP itself, the Djangocon EU talk to publicise it, the django-tasks implementation, and of course the django.tasks PR itself. The commit might have my name on it, but it's taken far more than just me to get us to where we are today, so I thought they'd be worth shouting out:

First off, the Mastodon post (or "toot" if you want to be technically correct) which started it all: from Tobi Degnon. Without it, django.tasks would almost certainly not exist today, instead being stuck in mostly fleshed out Google Doc I wrote 2 years prior.

For those who don't know, for a DEP (Django Enhancement Proposal) to start, it needs a "shepherd" - someone from the Django project to champion it, drive it, and navigate some of the internal processes. Carlton Gibson, who needs no introduction in the Django community, was that person. Carlton's passion for improving Django and belief in the vision is what kickstarted both mine and everyone else's enthusiasm in getting this started and still moving many months later.

The background tasks proposal originally started as a proposal to include them in Wagtail. At the time, I'd intended to choose a reasonably well supported background task system, and add it as a dependency to Wagtail. It wasn't until my colleague Tom Usher reviewed it and suggested the ability to switch out backends, rather than using a single implementation. I think without that 1 suggestion, this project would not exist.

Speaking of colleagues, my employer: Torchbox. For "letting" me use my personal development time to push this proposal forward, in all its formats. My productive times of times of the day shift around a lot, so being able to carve out a little time during the work day to focus on this helped it actually move. I'm sure from the commit history you can work out when I did that. Without this, I'd be stuck with evenings and weekends to work on this - and my free time there shrank quite a lot in late 2024.

Since announcing django-tasks at Djangocon Europe, 16 people have taken the time to contribute. Whether that's fixing small bugs, implementing the features they need, or just helping out with the open issues to take them off my plate. Open source is built on exactly these people, pushing it forwards.

When suggesting large scale changes to a popular project, you inherently invite opinions from a lot of people. Those enthusiastic for change, those longing for things to remain, and those who just want to give their 2p (or 2¢ in freedom units) to help someone out. The 3 months of review that the DEP received in the original PR were quite the rollercoaster for me - it was the first time my work had been seen and reviewed by so many people. Whilst 15 people added reviews, many more added comments and chimed in on the conversations. Some helped clarify my wording (I'm not the most articulate), whilst others spotted major issues with my approach.

354 days ago, I opened the first PR against Django to add the foundations of django-tasks. Since then, the project has changed quite a bit. At the time, the only documentation for django-tasks existed in my head, and a lot changed and solidified as I had to document how it was expected to work. A number of people have jumped in to contribute and review the fairly large PR. However I do want to call out @rtpg and @bluetech specifically for their in-depth reviews and especially helpful comments.

I've had a number of interesting conversations with people about django-tasks. Many at Djangocon Europe with people excited to see this finally coming to Django. But, when I started out, I'd never built anything like this before. A number of pub and ice-cream bar discussions with both Alistair Lynn and Peter Law made me realise the scale of what was in store for me. Not just building something to run Python functions, but something people are going to need to rely on, and the deterministic semantics which go along with it. Transactions in particular were a large topic of conversation, and how to handle those consistently for both database and non-database backends. These conversations eventually lead to the ENQUEUE_ON_COMMIT setting and semantics.

Whilst there are a number of task queues out there, especially in the Python ecosystem, RQ always stood out as my favourite. Having @selwin appear to give input on the RQ backend, and even make a few tweaks to RQ itself to make my life easier made me realise maybe humanity does stand a chance. Celery, django-lightweight-queue, django-db-queue, huey and django-q2 also all contributed a little inspiration to the project.

At Wagtail Space 2024 in the Netherlands, I started working on integrating django-tasks into Wagtail core, offloading a number of less-critical actions into the background. This work eventually released in Wagtail 6.4. There hasn't been much more offloading since then, but as Wagtail grows, the fact it can rely on being able to easily offload actions to the background will almost certainly enable new features. If you have a Wagtail site, and you've not yet enabled django-tasks, I strongly recommend it. Without Wagtail helping drive install numbers and users, django-tasks almost certainly wouldn't have the same level of reach or credibility it has today.

Django is an open source project, contributed to by a number of people, but 2 (now 3) people in particular have to do the grunt work to keep the project moving forward: the Fellows. Natalia and Jacob have helped massively pushing this project over the line, resulting in Natalia being able to push that all important "Merge" button. However, over the last few months, Sarah has had to endure a terrible fate: reading, reviewing, and trying to make sense of my code - I wouldn't usually wish that on anyone. The upstream django-tasks project looks, works, and is structured how I want it to be structured, based on my opinions. Having my code sit alongside the remaining 125k lines of Python which makes up Django only works if everything fits together properly, and is easily maintainable by anyone, not just me.

And of course, everyone who has given django-tasks a go. The network graph on GitHub shows quite a few installations (albeit inflated a little by Wagtail), but it's heartwarming to see so many installations and people using it. The fact a package I built is being used by EuroPython and the Python Software Foundation and so many more just feels crazy to me.

#What's next?

If you can, give the upcoming Django 6.0 Alpha release a go (once it's available). Not only will you get the new django.tasks, but you'll also receive a number of new Django niceties. If you can't, come give django-tasks a try. It's the same fundamental API, but can be easily swapped out once you upgrade to 6.0 with a simple find and replace.

django-tasks isn't going anywhere anytime soon. Just because the core is in Django 6.0, doesn't change its usefulness for those on 5.2 and older. By my maths, the repository won't retire before 30th April 2028 at the earliest - the end of Django 5.2 support, where all releases will have the foundations. django-tasks also acts as a playground - a place to develop and launch features without needing to go through Django's review process (whatever your opinions may be on it). It'll still be fully compatible with django.tasks, but if you want to get the newest features (like that shiny database backend), installing django-tasks is the way to do it. Similarly, new features will be added to django-tasks before they're added to upstream Django, and I'm determined to backport any improvements which are made upstream (and I'm already behind).

Moving forward, there's still a lot of work to do. The database backend exists, and is more than usable, but it's got a way to go before it's in the same league as the likes of Celery and RQ for everyone. There are a few resiliency improvements, but also big important features which a production-grade tasks system needs. The next one on my list is timeouts, which I have some ideas about, after which I'll need to tackle the monster that is retries. It might sound simple, but there's a lot of asterisks.

&LTaside>

My work on task timeouts is also the topic for my upcoming Pycon UK talk! If you're attending, be sure to come say "Hi"! If you're not, I'll publish the slides and recording as soon as they're available.

&LT/aside>

If you're interested in helping out with building the next iteration of django-tasks, or joining in the conversation, the GitHub repository is the place to go. Report a bug, fix an issue, vend opinions in discussions - everything is in the open for anyone to contribute to. If you have no idea what I'm on about (and yet have read this far), the docs for Django's new "Tasks framework" are worth a read.

What's in store for django.tasks in 6.1 - we'll see...