Python lazy imports you can use today – PythonTest

2 weeks ago 2

There’s a proposal, PEP 810 – Explicit lazy imports for Python to natively support lazy importing starting in Python 3.15.

However, it has not been accepted yet, and even if it is accepted, 3.15 is a year away. What do we do now?

The techniques covered in this post that allow you to use lazy importing NOW with Python 3.13, Python 3.12, …, really every version of Python.

Let’s look at a small code example that imports something.

mycode.py:

import somelib def main(): print("main() called") somelib.a_function() if __name__ == '__main__': main()

The module mycode.py needs to call a_function from somelib. In order to do so, mycode needs to import somelib.

We traditionally put imports at the top of a file at the top level (not in a function). The import somelib will do whatever Python does to import something, and end up with somelib as a name in the global scope of mycode.

Since the import is outside of a function, that importing happens at the time that mycode is imported.

Let’s throw some print statements into somelib:

somelib.py:

import time time.sleep(1) # Simulate a heavy import def a_function(): print("a_function called") # global scope print print("somelib imported")

Python imports code before running it

I know I’m stating the obvious. But I guess that’s about it. I’m stating the obvious. When we run the code with python mycode.py, Python will import the module, then run it.

$ python mycode.py somelib imported main() called a_function() called

See? somelib was imported before main() was called. Nothing exciting. Just setting the stage to shake things up.

All global level imports run upon import

If we don’t “run” mycode, but simply import it, somelib also gets imported, since the import is not in a function.
We can try this from the REPL or from the command line.

REPL:

>>> import mycode somelib imported

command line:

$ python -c 'import mycode' somelib imported

Why do I care about this? Because sometimes I want to import a module and NOT have all of it’s imports also get imported.

Why would I want to import something and NOT import that somethings imports?

Lots of reasons:

Startup speed. Especially if the import isn’t needed until something much later, like a user interaction, we can make code start zippier if we avoid importing slow import modules.
Seldom used imports. Some imports are there for not very often used circumstances. Better to just avoid the import until it’s needed.
Allowing a wide API without penalizing users of the API that just need a portion. Lots of packages do way more things than we need. Well designed packages split up functionality so that they only import part of their system as you need it.
Speeding up test collection. Tests import the code under test. And pytest test collection imports all of the test files before any test starts. So speeding up imports in test code can be especially helpful for large test suites.

I’m sure there are many other reasons as well. If you think of some I missed, let me know

Use lazy import by moving import into a function

We can move the import somelib out of the module global scope and into a function.

# import somelib def main(): print("main() called") import somelib somelib.a_function()

This causes Python to hold off on importing somelib until this function is called.

$ python mycode.py main() called somelib imported a_function() called

This has at some benefits:

somelib isn’t imported at mycode import time.
Anything somelib might import is also not imported right away.
We can speed up Python startup time by not importing stuff until needed.
somelib will end up never being imported if main() isn’t called.

Just import:

$ python -c 'import mycode'

Ok. That’s not very exciting. But note that “somelib imported” isn’t being printed.

REPL:

>>> import mycode >>> mycode.main() main() called somelib imported a_function() called

In the REPL we can really see that somelib doesn’t get imported when import mycode happens.

You can import something multiple times with no performance hit

If we need somelib in multiple functions, and we don’t know which will be called first, we can just import from every use. This effectively still imports once though, so you don’t get a performance hit.

mycode.py

def one(): print("one() called") import somelib somelib.a_function() def two(): print("two() called") import somelib somelib.a_function() def main(): one() two() if __name__ == '__main__': main()

Even though it looks like import somelib is done twice, it ends up only being one import. But it puts the name somelib in the local namespace of both functions one() and two().

$ python mycode.py one() called somelib imported a_function() called two() called a_function() called

You’ll notice that “somelib imported” only shows up once.

The reason we put it in both places is because the name somelib needs to be available in the local scope of both functions.

Keeping global scope with local import

If we know which function is always going to be called first, we can just import it at that time and set it to a global name

def one(): global somelib print("one() called") import somelib somelib.a_function() def two(): print("two() called") somelib.a_function()

This works just fine.

Using importlib.import_module() for lazy importing

You can also use importlib.import_module() to the same effect.

from importlib import import_module def one(): print("one() called") global somelib somelib = import_module("somelib") somelib.a_function() def two(): print("two() called") somelib.a_function() def main(): one() two() if __name__ == '__main__': main()

With import_module(), you can put it in multiple functions if you don’t want it in global scope. It will still only do the actual import once.

Using python -X importtime to find slow imports

I don’t usually start with lazy imports.
I usually start with any imports at the top of the file.

It’s when I’m optimizing startup time or test collection time that I look for which imports to switch to lazy loading.

To find which libraries are loading slowly, you can use Python’s builtin -X importtime.

$ python -X importtime mycode.py 2>&1 | sort -r | head -10 two() called somelib imported one() called import time: self [us] | cumulative | imported package import time: 1002021 | 1002021 | somelib import time: 1206 | 1206 | _collections_abc import time: 1095 | 4836 | site import time: 974 | 1882 | _frozen_importlib_external import time: 897 | 2327 | encodings import time: 718 | 2555 | os

This runs the code, but pays attention to how long it takes to import something. It reports cumulative import time, and it shows up in a kind of tree.

The report goes to stderr. So if you want to pipe it to something, keep that in mind.

In bash like terminals

2>&1 sends stderr to the stdout stream.
sort -r does a reverse sort. I want to see big numbers first.
head -10 shows the top 10 lines.

Speeding up test collection with lazy imports during testing

When writing test code, you have even more reasons to avoid slow imports, but also have a couple extra techniques available.

When pytest collects all of the tests it can see, it will import all test files during the collection phase, before running any fixtures or tests.

So if you have imports in global scope of test files, those imports will happen during test collection.

Let’s take a look at an example test.

test_somelib.py

import somelib def test_a_function(capsys): somelib.a_function() captured = capsys.readouterr() assert "a_function() called" in captured.out def test_something_else(): ...

With import somelib at the normal global namespace, somelib will be imported during test collection.

$ pytest -s test_somelib.py ============ test session starts ============ collecting ... somelib imported collected 2 items test_somelib.py .. ============= 2 passed in 1.05s =============

That “1.05s” is way too long, and we can see with -s allowing print statements that somelib is imported.

That’s not surprising. Just pointing it out.

This import happens during collection and even happens when we call the test that doesn’t need somelib.

$ pytest -s test_somelib.py::test_something_else ============ test session starts ============ collecting ... somelib imported collected 1 item test_somelib.py . ============= 1 passed in 1.05s =============

Still getting imported.

Moving import to test functions

Let’s use a technique we already have learned with straight Python and move the import into the test code function.

def test_a_function(capsys): import somelib somelib.a_function() captured = capsys.readouterr() assert "a_function() called" in captured.out def test_something_else(): ...

Now, when we call only test_something_else(), the import doesn’t happen.

$ pytest -s test_somelib.py::test_something_else ============ test session starts ============ collected 1 item test_somelib.py . ============= 1 passed in 0.04s =============

Using importlib with test functions

With imports in test code, we can use importlib.import_module() also.

from importlib import import_module def test_a_function(capsys): somelib = import_module('somelib') somelib.a_function() captured = capsys.readouterr() assert "a_function() called" in captured.out

Don’t use the global trick in test functions

We can’t just put global somelib in the first test functions and expect it to be there for other test functions. It might work when you try it, as we know pytest by default runs tests from top to bottom. However, with the ability for users to just call one test function, and randomize order and change order, and deselect tests, etc. you really can’t rely on the order of test functions being run.

However, we can stick imports in fixtures.

Moving import to a fixture

You can create fixtures for libraries you want to use, and just include them in the parameter list.

import pytest @pytest.fixture(name='somelib') def somelib_import(): global somelib import somelib return somelib def test_a_function(capsys, somelib): somelib.a_function() captured = capsys.readouterr() assert "a_function() called" in captured.out def test_something_else(): ...

This works fine, but looks weird.
It looks a little less weird with importlib.

import pytest from importlib import import_module @pytest.fixture() def somelib(): _somelib = import_module('somelib') return _somelib def test_a_function(capsys, somelib): somelib.a_function() ...

Either of these methods works to speed up test collection and avoid imports for test methods that don’t need them.

Moving import to an autouse fixture

If everything in the test file (or nearly so) needs the import, we can make the fixture autouse and set the module name to global. Then we don’t have to name it in each test.

import pytest @pytest.fixture(autouse=True) def somelib_import(): global somelib import somelib def test_a_function(capsys): somelib.a_function() captured = capsys.readouterr() assert "a_function() called" in captured.out def test_something_else(): ...

Setting up an “autouse” import doesn’t save us the import time for test_something_else anymore, so I reserve this technique for test modules that all use the import.

$ pytest -s test_somelib.py::test_something_else ============ test session starts ============ collected 1 item test_somelib.py somelib imported . ============= 1 passed in 1.05s =============

However, when we run other test files, we still get the test collection speedup.

$ pytest -s -k foo ============ test session starts ============ collected 3 items / 2 deselected / 1 selected test_foo.py . ====== 1 passed, 2 deselected in 0.04s ======

Feedback welcome

These are techniques I use. If you have other, I’d love to hear them. Feel free to contact me

Read Entire Article