Modern Python CI with Coverage in 2025

8 hours ago 1

November 03, 2025 | View Comments

/media/slayer.png

Warning

This blog post has been written by an LLM for the most part. The image above has been generated by a soulless ghoul called Gemini 2.5. I hope you still find the blog useful!

I've recently revisited building a GitHub CI pipeline for a Python project that includes coverage reporting, with only free-as-in-beer tooling and I've landed in a pretty nice place.

The most interesting parts of the toolchain in this set up are:

py-cov-action: Coverage reporting without external services like Codecov or Coveralls
pytest-xdist: Parallel test execution using all available CPU cores
uv: Package management that's 10-100× faster than pip

The integration of these tools requires attention to some nuances that aren't very well documented elsewhere, and so while each individual tool is documented very well, the interplay can get a bit tricky, so that's what we're focusing on in this blog.

Six critical gotchas

These silent failures can potentially waste you a couple of hours. Each one will make your CI look successful while coverage is actually broken.

Gotcha #1: Using coverage run -m pytest with xdist

Symptom: Coverage shows 0% or drastically low percentages.

Root cause: coverage run -m pytest -n auto bypasses pytest-cov's xdist support. Coverage.py only sees the main process, not the workers.

The fix: Always use pytest -n auto --cov

Gotcha #2: Missing relative_files = true

Symptom: Coverage works locally but shows 0% in CI.

Root cause: Coverage.py uses absolute paths by default (/home/runner/work/myproject/myproject/file.py). These don't match GitHub's file structure.

The fix in pyproject.toml:

[tool.coverage.run] relative_files = true

Without this, py-cov-action can't map coverage data to source files.

Gotcha #5: Missing pytest-cov plugin

Symptom: pytest: error: unrecognized arguments: --cov

Root cause: pytest-cov not installed.

The fix in pyproject.toml:

[tool.uv] dev-dependencies = [ "pytest-cov>=6.0.0", ]

Gotcha #6: E2E tests with subprocesses contribute 0% coverage

Symptom: E2E tests with Playwright or Selenium pass but show 0% coverage for server code.

Root cause: Tests spawn a subprocess. Coverage.py in the parent process can't measure the child.

The problem in practice:

# tests/test_frontend_e2e.py @pytest.fixture def live_server(): process = subprocess.Popen( ["uv", "run", "mypackage", "serve"], # Separate process! stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) # ... wait for server to start ... yield server_url process.terminate() # Server shuts down def test_frontend_loads(live_server): # This test passes but contributes 0% coverage response = requests.get(live_server) assert response.status_code == 200

The fix Set COVERAGE_PROCESS_START environment variable and invoke via coverage run -m:

@pytest.fixture def live_server(): env = os.environ.copy() env["COVERAGE_PROCESS_START"] = "pyproject.toml" # Copy pytest-cov environment variables for key, value in os.environ.items(): if key.startswith("COV_"): env[key] = value # Invoke via coverage run process = subprocess.Popen( ["uv", "run", "coverage", "run", "-m", "mypackage", "serve"], env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, )

However, you should take a step back and think if you want E2E tests to play into coverage. Some might say that this will lead to bloated coverage numbers, but it's also nice to look at E2E test coverage in isolation. However, beware that any frontend code written in a language other than Python will not be tracked.

Some reasons why you might not to include coverage for E2E tests are:

Unit tests already cover most backend logic directly
Integration tests already hit the same API endpoints
Coverage.py only measures Python code, not JavaScript
E2E tests primarily verify frontend/backend integration

Complete working example

Here's a production-ready setup you can copy and adapt. These files work together to provide parallel testing, coverage reporting, and fork-safe PR comments.

Note

This example uses the two-workflow pattern for fork PR support described earlier.

.github/workflows/ci.yml

name: CI on: pull_request: push: branches: [main, master] permissions: contents: write pull-requests: write checks: write actions: read jobs: test: name: Test & Coverage runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Install uv uses: astral-sh/setup-uv@v6 - name: Install Python 3.14 run: uv python install 3.14 - name: Sync dependencies run: uv sync --dev - name: Install Playwright browsers (if needed) run: uv run playwright install chromium # Remove this step if you don't have E2E tests - name: Run tests with coverage run: | uv run pytest -n auto -v \ --cov --cov-report=xml --cov-report=html --cov-report=term \ --junitxml=test-results.xml - name: Upload coverage artifacts uses: actions/upload-artifact@v4 if: always() with: name: coverage-report path: | .coverage coverage.xml htmlcov/ include-hidden-files: true # Critical for .coverage file - name: Publish test results uses: dorny/test-reporter@v1 if: always() with: name: Test Results path: test-results.xml reporter: java-junit - name: Coverage analysis id: cov uses: py-cov-action/python-coverage-comment-action@v3 with: GITHUB_TOKEN: ${{ github.token }} ANNOTATE_MISSING_LINES: true ANNOTATION_TYPE: warning - name: Store PR comment if: steps.cov.outputs.COMMENT_FILE_WRITTEN == 'true' uses: actions/upload-artifact@v4 with: name: python-coverage-comment-action path: python-coverage-comment-action.txt

pyproject.toml configuration

[project] name = "mypackage" version = "0.1" requires-python = ">=3.14" [tool.uv] package = true dev-dependencies = [ "pytest>=8.0.0", "pytest-xdist>=3.6.0", "pytest-cov>=6.0.0", "mypy>=1.13.0", "ruff>=0.8.0", ] [tool.coverage.run] source = ["mypackage"] omit = [ "tests/*", "*/__init__.py", "*/conftest.py", ] relative_files = true # Required for CI! [tool.coverage.report] exclude_lines = [ "pragma: no cover", "def __repr__", "def __str__", "raise AssertionError", "raise NotImplementedError", "if __name__ == .__main__.:", "if TYPE_CHECKING:", ] show_missing = true precision = 1 [tool.pytest.ini_options] testpaths = ["tests"] addopts = "-v"

Verifying your setup

After pushing these files, here's how to verify everything works:

Check relative paths

Run tests locally and inspect coverage.xml:

grep 'filename=' coverage.xml | head -3

You should see relative paths like filename="mypackage/cli.py", not absolute paths like /home/runner/work/....

Verify artifact upload

In GitHub Actions:

Go to your workflow run
Scroll to "Artifacts"
Download coverage-report
Verify .coverage file exists inside

What success looks like

Locally:

$ uv run pytest -n auto --cov --cov-report=term ============================= test session starts ============================== ... ======================= 231 passed, 2 skipped in 61.14s ======================== Name Stmts Miss Cover ------------------------------------------- mypackage/api.py 139 23 83.5% mypackage/cli.py 397 221 44.3% ... TOTAL 1427 488 65.8%

In GitHub Actions, you'll see:

Test Results check with pass/fail counts
Coverage comment on PR with diff
Line-by-line annotations on changed files
Badge in python-coverage-comment-action-data branch

Migration notes

From Codecov/Coveralls

Replace your codecov step:

# Old - uses: codecov/codecov-action@v3 with: token: ${{ secrets.CODECOV_TOKEN }} # New - uses: py-cov-action/python-coverage-comment-action@v3 with: GITHUB_TOKEN: ${{ github.token }}

No external token needed. Badge URL changes from codecov.io/gh/USER/REPO/badge.svg to github.com/USER/REPO/raw/python-coverage-comment-action-data/BADGE.svg.

From pip to uv

Replace in your workflow:

# Old - uses: actions/setup-python@v4 with: python-version: "3.14" - run: pip install -r requirements.txt # New - uses: astral-sh/setup-uv@v6 - run: uv python install 3.14 - run: uv sync --dev

Create pyproject.toml with your dependencies and run uv lock to generate the lockfile.

Next steps

Copy the workflow files above
Add relative_files = true to your pyproject.toml
Push to GitHub and watch CI run
Add the badge to your README from the data branch

For slow tests, mark them with @pytest.mark.slow and run them separately. For coverage gaps, focus on unit tests for business logic.

Read Entire Article