Reinventing the Python Wheel

We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.

It is no secret that the Python packaging world is at something of a crossroads; there have been debates and discussions about the packaging landscape that started long before our 2023 series describing some of the difficulties. There has been progress since then—and incremental improvements all along, in truth—but a new initiative is looking to overhaul packaging for the language. At PyCon US 2025, Barry Warsaw and Jonathan Dekhtiar gave a presentation on the WheelNext project, which is a community effort that aims to improve the experience for users and providers of Python packages while also working with toolmakers and other parts of the ecosystem to "reinvent the wheel". While the project's name refers to Python's wheel binary distribution format, its goals stretch much further than simply the format.

Warsaw started things off by noting that, while he and Dekhtiar both work for NVIDIA, WheelNext is a "community-driven initiative that spans all of the entire Python community". He put up profile pictures from around 30 different people who had already been contributing to the WheelNext GitHub repository; "it's really open to anybody", Warsaw said.

Before getting into the meat of WheelNext, it is important to "celebrate the wins for the Python packaging community", he said, showing some screen shots that had been taken a few weeks before the talk in May. The numbers of projects (600K+), releases (nearly 7 million), files (14.1 million), and users (920K+) listed on the Python Package Index (PyPI) home page are eye-opening, for example. The PyPI Stats page showed numbers that "certainly blew me away; 1.6 billion downloads a day [on that day], 20 billion downloads a month". He showed some other graphs that illustrated the "prodigious amount of data and packages that are being vended by PyPI".

It is clear that Python packages, and wheels in particular, have been extremely successful. They are used "every day, in many many different ecosystems, in corners of the Python world that we're not even aware of" and that is the result of a lot of hard work over decades by many people, organizations, and companies. "Wheels pretty much serve the needs of most users most of the time ... so that's awesome." Over time, though, as Python reaches more development communities and additional use cases arise, "cracks are beginning to show"

WheelNext

His elevator pitch for WheelNext: "An incubator for thinking about the problems and solutions of the packaging community for the next, let's say, ten years, five years, 30 years". WheelNext goes well beyond just the wheel format; "we're really talking about evolving the Python-packaging ecosystem". Among the companies, organizations, and individuals involved are all of the different stakeholders for packaging, beyond just users: "tool makers, environment managers, installers, package managers", many tools that consume wheels, people building packages for PyPI, and so on. All of those are affected by what is working well—and not so well—for packaging.

He showed a slide of the logos of a dozen or so communities and projects that are part of the effort, noting that it was just a sampling of them. He said that they were mostly in the scientific-computing part of the Python world; "that 's just because I think the packaging ecosystem doesn't serve their needs quite as well as many other communities".

A lot of the inspiration for WheelNext came out of the pypackaging-native web site; he recommended visiting that site for "really excellent, detailed information about the problem space that we're trying to solve". WheelNext is the other side of the coin, trying to find solutions for the problems outlined there.

For example, wheels and tools like pip are not set up to handle GPUs, CPU micro-architectures, and specialized libraries (e.g. for linear algebra); in those cases, users want a wheel that targets specific versions of those variables. Limits on the size of wheels that are imposed by PyPI need to be addressed as part of WheelNext. In addition, there are native dependencies: libraries written in C, C++, Rust, or other languages that are needed by Python modules, along with any dependencies. It is difficult for users to specify exactly what they need in those cases.

The WheelNext project has a set of axioms that it has adopted as part of its philosophy. "If it works for you now, great, it'll continue to work for you later"; the project is not looking to change things for parts of the community where the status quo already works. Beyond that, the project is prioritizing the user experience of package installation and trying to push any complexity into the tools. WheelNext does not want to create another silo or its own ecosystem, it wants to meet users where they are today; users already have tools that they like, so it does not make sense to force them to learn another.

The idea is for WheelNext to come up with ecosystem-wide solutions, not ones that only work for a single tool or service. For example, there are "lots of third-party indexes that exist" beyond just PyPI, Warsaw said; the intent is to think about "how are we going to standardize what we need to standardize for interoperability". Backward compatibility will be prioritized, but if there comes a need to break that to improve things, the plan is "to do so intentionally and explicitly with a very defined migration path".

Problems

At that point, Dekhtiar took the stage to further discuss the problems that WheelNext is trying to address. First up was the PyPI file-size limitations; by default PyPI files can be up to 100MB in size; that can be raised to 1GB by PyPI staff, which is a manual process, thus kind of painful. Meanwhile, a project is limited to 10GB in total size for all of its files; "you are kind of boxed in by multiple limits at the same time". This problem is particularly acute for scientific projects and those shipping large AI models.

One possible solution to that is for those who need to distribute larger objects to have their own indexes. "That does not work as well as we would wish", in part because pip's interface is painful to use with extra indexes. In addition, there can be security problems because multiple indexes can provide packages with the same names, some of which could be buggy or malicious. The dependency resolver will try to choose the best version to install based on version numbers, which can be set by attackers; "in terms of security, it is difficult to manage". So, improving the mechanism for resolving and prioritizing indexes is an important target for WheelNext.

As Warsaw had noted, backward compatibility is a major goal for the initiative; the intent is to fix the problems "without reinventing the workflow", Dekhtiar said. It is difficult to do so using the existing wheel format because it cannot be extended in a backward-compatible way; there is no way to express that older versions of pip should see one thing, while newer versions should see a different format. That lack is holding up the ability to add support for symbolic links, Zstandard compression (which would help with PyPI file-size limitations somewhat), and METADATA.json files (which would help implementing Python dependency lock files and package resolution).

Many Python scientific-computing packages rely on binary libraries of various sorts. A given application might use multiple packages, each of which relies on (and ships) its own version of the OpenBLAS linear-algebra library. Being able to share common libraries between packages in an efficient manner would reduce the number of redundant libraries that are shipped in packages and loaded at run time.

Some packages support multiple options for backends that provide visualization tools, web servers, or GUI toolkits, for example. They often require that one of the options is chosen when the package is installed, but users may not have a opinion about which they want. Like Python does with default arguments to functions, he said, it would be nice to have a way to specify a default "extra" that will get installed if no other choice is made.

Right now, wheels are identified by a set of platform identifiers that do not include all of the different possibilities. In particular, packages can be built and optimized for specialized hardware, such as GPUs, FPGAs, CPU micro-architectures, specialized instruction sets (e.g. AVX512), and so on, but there is no mechanism to select wheels based on that criteria. Without fine-grained selection, "what you end up having is the lowest common denominator, which is optimized for nobody", Dekhtiar said.

The problem has been "solved" for some projects by having a web-site selector that allows users to choose the right package, but it forces them to read the documentation and set up a different index for getting their packages. "This is awesome, because it allows us to do what we need", but "we are a little bit sad that this is the best answer we have today and we wish we could do better".

Since the talk was 30 minutes in length, he said, they could not cover the entirety of WheelNext, but he wanted to quickly go through some of the PEPs that have been discussed along the way. He started with the (withdrawn) PEP 759 ("External Wheel Hosting"); it is a proposed solution for the problems with multiple repositories (indexes) for Python artifacts. PEP 771 ("Default Extras for Python Software") is meant to address the need for specifying default backends as he had described earlier. He said that PEP 777 ("How to Re-invent the Wheel") was meant to help ensure that the wheel format evolves in ways that would not require backward-incompatible changes in the future. PEP 778 ("Supporting Symlinks in Wheels"), which was recently deferred, is challenging because not all filesystems support symbolic links, but there is a need to share libraries as he mentioned earlier. The build isolation passthrough PEP does not have a number, but it is meant to help in building Python extensions based on experimental or development versions of packages on the local machine.

Governance

With that, Warsaw stepped up to talk about packaging governance and PEP 772 ("Packaging Council governance process") in particular. Over the years, in the Python community, "there's been as little bureaucracy as we could possibly get away with and more of a grassroots movement for handling things". As it becomes clear "that we need some more formalism, we figure out how to do that"; the creation of the Python steering council is a good example of how that works.

The community has recognized a need for some more formalism in packaging governance recently. There are essentially two developers who each "have a vertical slice of the packaging ecosystem and they have standing delegations from the Python steering council" to decide on packaging PEPs, Warsaw, who is a member of the steering council, said. There are concerns about the "bus factor", but having people in that situation "also means that there is a lot of burden on that one person to do everything and make sure that they get it right" for their slice.

So the PEP is an effort to bring the steering-council model ("which is mostly successful") to the packaging community. The idea is that the steering council can delegate packaging decisions to a council that is elected by the large community of packaging stakeholders. Those who are familiar with the workings of the steering council will find the election of the packaging council and its operation to be similar. There are some differences, due to the nature of the packaging community; currently the main effort is to define the voting community that will vote on the five members of the packaging council. His hope was that the PEP could go to the steering council for a vote soon and that the packaging council could get started sometime this year; an updated version of the PEP was announced shortly after PyCon.

More PEP

Dekhtiar returned to the podium to talk about more WheelNext initiatives; PEP 766 ("Explicit Priority Choices Among Multiple Indexes") was up first. As he had said, the pip interface for using multiple indexes is cumbersome; it would be better if users could specify "PyTorch comes from here, NumPy comes from there, and the CUDA wheels come from there". The interface needs work, but there is also a need to protect against security problems when the installers are choosing packages based on their origin and version numbers. PEP 766 is more meant "to define the vocabulary, the wording, than actually behavior"; the intent is to have common language for the installers to use when describing their resolution behavior with respect to multiple indexes.

Sharing binary files, like OpenBLAS, between wheels or with system-installed libraries (if they are present) is difficult; "there is no safe way to do that that is common and standardized across the ecosystem". The WheelNext participants want to find a solution for a native library loader that is a kind of "best practice" approach to the problem, which can be shared throughout the community. He likened it to importlib, but one "that's specific around loading binaries". There is a saying in the WheelNext community, he said, "'good enough is better than nothing at all' and right now we have nothing at all" for handling shared libraries.

Wheel variants is another potential PEP. Dekhtiar said that he wanted to share the dream that WheelNext participants have about the next iteration of the wheel format. Today, the platform that is associated with a wheel includes the Python ABI version (e.g. 3.12, 3.13, or 3.13 free-threaded), the operating system, the C library (e.g. manylinux for the GNU C library), and the CPU architecture. Those are encoded into the name of the wheel file. Those tags are not sufficient to describe all of the platforms in use, but "constantly adding tags to better describe your platform is not a scalable practice". There are different GPUs, application-specific integrated circuits (ASICs), and, some day, quantum-computing devices; even if the community wanted to fully describe today's systems, "we don't have the language to be able to do that".

Instead, the idea of wheel variants is to have the installer determine what the local system has installed, then use that information to choose the right wheel. For example, for JAX and PyTorch, the installer could determine which version of CUDA is installed, what kind of Tensor Processing Unit (TPU) there is, and which instructions are supported by the CPU, then it can "pick the best". He went through some scenarios using a prototype pip that would download vendor plugins to detect various aspects of the environment (CPU micro-architecture or CUDA version, for example). From a combination of the package metadata and the running system, it would determine which wheels to request for installation. At the time of the talk, the prototype worked with a subset of packages and just pip as an installer, but the hope is to get it working with others in order to collect more feedback.

Conclusion

Warsaw finished the presentation with a "call to action", inviting people to get involved with WheelNext and to bring their use cases. The project has various ways to participate and is actively seeking feedback and contributions. For interested readers, the YouTube video of the talk is also available.

[Thanks to the Linux Foundation for its travel sponsorship that allowed me to travel to Pittsburgh for PyCon US.]