Recent Disruptive Changes from Setuptools

In late March, version 78.0.1 of Setuptools — an important Python packaging tool — was released. It was scarcely half an hour before the first bug report came in, and it quickly became clear that the change was far more disruptive than anticipated. Within only about five hours 78.0.2 was published to roll back the change, and multiple discussions were started about how to limit the damage caused by future breaking changes. Nevertheless, many users still felt the response was inadequate. Some previous Setuptools releases have also caused problems on a smaller but still notable scale, and hopefully the developers will be more cautious going forward. But there are also lessons here for the developers of Python package installers, ordinary Python developers and end users, and even Linux distribution maintainers.

Python packaging

Python code is commonly distributed using a Python-native packaging system, rather than as a Linux package or standalone executable. The basic ideas have been described by LWN before.

Packages are available in two standard formats: the "wheel" format, and the "sdist" (or source distribution) format. Both are compressed archives, but a wheel has been pre-built and can be installed without much work besides copying files. An sdist, on the other hand, roughly represents the developer's source tree (although it might omit tests, documentation, etc.), and must be built on the end user's machine. This is generally only useful for packages that contain non-Python code, but it allows for additional setup prior to installation. An sdist also includes instructions (perhaps including Python code that will be automatically run) to compile non-Python code, arrange files, and ultimately "build" a corresponding wheel.

Creating packages — in either format — can be quite complex. Usually, therefore, most of the work is delegated to a build backend, which provides logic to create an sdist or wheel directly from a source repository, as well as to create a wheel from an sdist. To install an sdist, an installer such as pip will check the sdist's metadata to determine the right build backend, ensure that backend is installed, and invoke it via a standard API. The resulting wheel is then installed as if it had been downloaded directly.

By default, this process uses a kind of build isolation: the backend is installed in an isolated, temporary virtual environment. This ensures that builds don't interfere with each other, allows different packages to use different versions of Setuptools, and means that Setuptools doesn't need to be installed ahead of time in pip's own environment.

Historically — since back when end users were expected to resolve and install dependencies themselves — Setuptools was the standard tool for a variety of development tasks. Nowadays, much of its functionality has been deprecated, but it still acts as a build backend — and for legacy support reasons, packages are assumed to use it by default.

What went wrong

Projects built with Setuptools may describe some of their metadata in a configuration file named setup.cfg (which uses a simple INI-like format, with key-value pairs organized into sections). The unreleased Setuptools 78.0.0 added stricter validation for the contents of such files. Historically, hyphens and underscores were treated as equivalent in the names of keys appearing in the file — behavior originally implemented by the distutils standard library module. In 2018, a bug report claimed that this normalization caused problems with tox (a Python tool for automating development tasks such as testing). Setuptools developer Jason R. Coombs agreed that there was a problem; after some changes to the Setuptools configuration-parsing code, a a deprecation warning for hyphens in key names was added in 2021.

Setuptools version 78 sought to turn that warning into an error. Developers creating new sdists would be able to fix the problem trivially, but the change also immediately broke the automated wheel-building process for many sdists already available on the Python Package Index (PyPI).

In fact, this change was already known to be incompatible with the popular Requests library — its setup.cfg includes provides-extra and requires-dist keys, where the standard names are provides_extra and requires_dist respectively. But, since Requests uses only Python code, and already publishes a wheel, it would only cause problems for those (such as Linux distribution maintainers) who insist on building directly from source.

The 78.0.1 patch simply removed Requests from the Setuptools integration tests. Setuptools developer Anderson Bravalheri also submitted a pull request to Requests to fix the problem with setup.cfg (though as it turns out, the next minor version of Requests is expected to modernize its packaging setup completely, so as to avoid any future issues). However, packages with now-invalid setup.cfg files turned out to be much more common than expected — about 12,000 packages are estimated to have been affected.

Reporting and updating

When a resulting installation failure was first reported, Bravalheri's initial response was fairly dismissive:

Hi @andy-maier, please contact the package developers and inform about the problem highlighted in the error message: [...] This [key name in setup.cfg] has been deprecated in 2021.

He followed that up with:

We bumped the major version of setuptools from 77.0.3 to 78.0.0 to indicate an intentional breaking change, namely the removal for the deprecated handling of --separated options in setup.cfg. They are described in this section: https://setuptools.pypa.io/en/latest/history.html#deprecations-and-removals.

At some level, this response makes sense. The project correctly followed its semantic versioning policy, and the deprecation had occurred years ago. In principle, users could avoid problems by simply not upgrading to the new Setuptools version, and developers could patch their setup.cfg files to meet the new expectation. However, things aren't really that simple. Updating the released packages is a lot of work for everyone — it's clear that many maintainers also resent this kind of churn. Besides, it doesn't prevent attempts to install the previous versions of the package with now-invalid setup.cfg files.

On the user side, meanwhile, build isolation makes it difficult to keep Setuptools downgraded — because of all the temporary copies installed automatically. Historically, there was no good way for packages to specify what version of Setuptools they used. That's possible now, but projects using Setuptools are under no pressure to change anything. And the standards-mandated default is to use the latest available version. The net result is that every sdist following any kind of deprecated practice becomes a ticking time bomb.

Worse yet, every package that breaks has the potential to cause major ripple effects, in today's world of "software ecosystems" where projects often have large graphs of transitive dependencies. Some of the projects broken this time around turned out to be abandoned or poorly maintained, but used by many others. For example, the stringcase module broke due to the change, but it is seemingly abandoned; when I learned of this, I decided to republish the library following modern packaging standards.

Similar events

This is not the first time that a breaking change in a major Setuptools version was disruptive. In recent memory, arguably the most similar case is that of Setuptools version 72, which attempted to remove the setuptools.command.test module. This, too, was following up on a long-standing deprecation of using setup.py directly to run tests. This isn't even relevant to installing packages from PyPI — but because setup.py is Python code, removing the module causes an ImportError at build time. Notably, the Requests project was also impacted by this, even though its developers were no longer even using this workflow.

Less notably, Setuptools version 77 was intended to enforce standards for how license files are referred to in project metadata, while also implementing the new standard enabling the use of SPDX license expressions in package metadata. However, the implementation caused problems for some large multi-language projects (notably for Apache projects) where the source tree has relevant license files in a parent directory of the actual Python project. Setuptools version 71 changed its vendoring system to favor separately-installed versions of its dependencies (as a stepping stone toward de-vendoring); this caused problems for users when those dependencies removed functionality that was still present in the vendored version. An internal reorganization in version 69 broke Astropy's continuous-integration (CI) system, and another in version 70 broke PyTorch. Not every new version causes widespread problems, but it's certainly possible to continue this list.

Criticism

Aside from the GitHub issue, users critiqued handling of the incident on Hacker News and Reddit. In particular, many questioned why the use of hyphens should have been deprecated in the first place. Indeed, one might infer that Setuptools can't really ever safely remove support for anything — as much as the developers might like to clean up the code, Hyrum's Law reigns supreme. Setuptools is seen as critical infrastructure for Python, and it's clear that the Python world will be stuck with old packages following deprecated packaging practices for quite some time.

The problematic version 78.0.1 was also never deleted nor "yanked" (to prevent pip from installing it unless explicitly selected by an exact version number) from PyPI, causing further objections.

One might argue that the simple fact that Setuptools is on version 78 points to a problem in itself — that Setuptools development is moving far too quickly. This is not entirely fair: about half of Setuptools major versions date to before the implementation of the new pyproject.toml-based system, with version 39 being released in March 2018. However, we still now see about five new major versions of Setuptools per year. Progress on updating legacy projects to modern standards, needless to say, has been far slower. While I'd prefer not to see Setuptools development artificially slowed, it would be nice to have breaking changes bundled together so that they occur less often.

Discussion

More importantly, though, there needs to be better awareness of such changes in Setuptools, and better processes for addressing them. On the official Python discussion forums, pip developer Damian Shaw started a thread about how build backends could improve their approach to deprecating and removing functionality, summarizing the overall problem for the Python ecosystem:

In this ecosystem it has meant that any build backend that is heavily relied on that makes a backwards incompatible change can break user workflows on a very large scale, and the tooling available for users to recover isn't well developed.

The ensuing discussion revealed that the documentation for some other build backends recommends setting an upper bound for the backend version in the project metadata. However, doing so can potentially break packages in the future — for example, if a newer version of the backend is required to work under a new version of Python.

Another approach is to have the installer use different logic to choose a build backend version. Paul Moore, a pip developer, initially resisted such a change:

I don't think it's inevitable, and the UX [user experience] design is one of the things I'm concerned about. IMO, pip shouldn't be taking on complexity here - I don't see any evidence that this is a major issue. We've never had any problems like this with any other build backend. And while I don't want to blame setuptools, I also don't want pip's feature set to be dictated by mishaps in setuptools' release management.

However, the discussion on the initial Setuptools bug report revealed that pip already provides some useful functionality here: a hack that allows end users to constrain the version of the build backend used for isolated builds. The pip team then discussed improving the surrounding user interface and making this an officially blessed approach to the problem. An analogous approach is explicitly supported by uv; Poetry users installing dependencies in a development environment may have to wait for the resolution of a GitHub issue. Pip users can also just disable build isolation, which might fix the problem in some cases.

Regarding front-end tools failing to expose warnings, Bravalheri proposed that installers should show build warnings to end users by default. This is not the first time the problem of deprecations has been raised for the language. Python developer Steve Dower noted that end users can't really do anything about the warnings. However, if a project only publishes an sdist, the maintainers won't see the output from trying to build a wheel unless they test installation locally. Even if end users can't fix the problem, they could issue bug reports about the warnings if they are shown.

A final problem with communication between installers and build backends is that the existing API doesn't seem to provide a good way to separate warnings from the rest of the build output. Moore suggested building a separate system for this:

So why don't we ask the setuptools maintainers what they would like from build frontends? Not right now - emotions are too high at the moment - but once things have cooled down, we should work with setuptools to understand how we can help things go more smoothly in future.

He had a list with several suggested paths for working with Setuptools developers, including looking into ways to get deprecation warnings in front of users:

There was no way for setuptools to get the deprecation warning in front of their users? Then frontends could add a way for backends to issue "priority warnings", which will be displayed even when normal output is suppressed.

Hopefully, something like this will be implemented in the future.

In summary, the Python packaging system is amazingly complex — to the point that a quibble over hyphens versus underscores can break thousands of packages and lead to hundreds of discussion posts. But with this complexity comes opportunity for many parties to improve the experience.

---