Fixture "not found" on discontinuous CLI ordering of test module paths

[andrew-meter]

Disclaimer:
After hitting this problem, I used Claude to assist in pinning down root-cause hypothesis and to synthesize a self-contained repro script demonstrating working vs non-working CLI ordering and some proposed solutions.

I've written this bug report myself, but armed with insights gained using Claude.

Versions / platform:

• Linux (docker image)
• pytest 9.1.1
• Python 3.14.6

pip list:

$ pip list | grep -i pytest
allure-pytest         2.16.0
pytest                9.1.1
pytest-asyncio        1.4.0
pytest-base-url       2.1.0
pytest-dependency     0.6.1
pytest-order          1.5.0
pytest-playwright     0.8.0
pytest-rerunfailures  16.3
pytest-testconfig     0.2.0

I'm hitting an issue where test modules invoked with pytest can fail to resolve their fixtures based on a discontinuous ordering of module paths at the CLI.

This occurs when a fixture is provided through a conftest file that resides at an intermediate location in the directory hierarchy of the involved test modules.

Our use case is within a CI workflow in Github:
We have a workflow that as part of merge-gating, any of our pytest module files that are edited in a PR will get executed and the result must be successful in order for the PR to be submitted.

This workflow can end up passing a list of module paths to pytest in a non-sorted order.

FAILS:
pytest tests/woo/assignment/test_a.py tests/test_resume.py tests/woo/assignment/test_b.py

WORKS:
pytest tests/test_resume.py tests/woo/assignment/test_a.py tests/woo/assignment/test_b.py

A fixture named shared resides at tests/woo/conftest.py .

At the CLI, if the input list of test modules is not sorted so modules in the same directory tree are in consecutive order, the noted fixture can fail to be found and pytest surfaces error:

E       fixture 'shared' not found

Using Claude to debug the failure, eventually it surfaced what's suspected as the root-cause:
Two DIFFERENT 'tests/woo' Dir node objects are created through _pytest.fixtures.FixtureManager._matchfactories when the CLI module ordering is discontinuous with respect to module paths.

It also shows a proposed patch to _matchfactories which "fixes" the issue. The given description of the fix is:

make fixture matching fall back to baseid (string) matching when node-identity matching fails.
(this description is from Claude)

I can confirm that the fix at least works in the self-contained limited example. I will admit I don't know enough of pytest internals to confidently state it would be a uniform and fully generic solution.

The self-contained repro script (bash) is below. Happy to share any other info!

Repro script:

#!/usr/bin/env bash
#
# Minimal, self-contained reproduction of the pytest "shared conftest fixture
# becomes 'not found' depending on command-line file order" bug.
#
# Pure pytest -- NO internal code, NO plugins, NO __init__.py required.
# Requires pytest >= 8.0 (the Directory-node fixture-scoping rework).
# Reproduced on pytest 9.1.0 AND 9.1.1.
#
# Usage:  bash pytest_repro_sh
#
set -u

# ---------------------------------------------------------------------------
# Build a throwaway test tree:
#
#   <tmp>/
#     pytest.ini
#     tests/
#       test_resume.py            <- a module DIRECTLY under tests/ (sibling of woo/)
#       woo/
#         conftest.py              <- defines the shared fixture `shared`
#         assignment/
#           test_a.py              <- uses `shared`
#           test_b.py              <- uses `shared`
#
# The fixture lives one level ABOVE the tests (tests/woo/conftest.py); the two
# tests that use it live in the SAME leaf dir (tests/woo/assignment/).
# ---------------------------------------------------------------------------
ROOT="$(mktemp -d)"
trap 'rm -rf "$ROOT"' EXIT
cd "$ROOT"

mkdir -p tests/woo/assignment
printf '[pytest]\n'                                              > pytest.ini
printf 'import pytest\n@pytest.fixture\ndef shared():\n    return 1\n' > tests/woo/conftest.py
printf 'def test_a(shared):\n    assert shared == 1\n'           > tests/woo/assignment/test_a.py
printf 'def test_b(shared):\n    assert shared == 1\n'           > tests/woo/assignment/test_b.py
printf 'def test_resume():\n    assert True\n'                  > tests/test_resume.py

# A tiny plugin that demonstrates the one-line-idea UPSTREAM fix: make fixture
# matching fall back to baseid (string) matching when node-identity matching
# fails. (pytest's _matchfactories yields on object identity only.)
cat > fix_matchfactories.py <<'PYEOF'
import _pytest.fixtures as F

def _matchfactories(self, fixturedefs, node):
    parent_nodes = set(node.iter_parents())
    parentnodeids = {n.nodeid for n in parent_nodes}
    for fd in fixturedefs:
        # FIX: accept either node-identity match OR baseid (string) match.
        # Upstream only does `fd.node in parent_nodes` when fd.node is set,
        # which fails when collection built a DUPLICATE Dir node object.
        if (fd.node is not None and fd.node in parent_nodes) or fd.baseid in parentnodeids:
            yield fd

F.FixtureManager._matchfactories = _matchfactories
PYEOF

PY="python -m pytest -q -p no:cacheprovider"
A="tests/woo/assignment/test_a.py"
B="tests/woo/assignment/test_b.py"
R="tests/test_resume.py"

hr() { printf '\n========================================================================\n'; }

echo "pytest version: $(python -m pytest --version 2>&1 | head -1)"
echo "tmp tree: $ROOT"

hr
echo "CASE 1  BROKEN  (interleaved: A, resume, B  -> the order CI's selection produced)"
echo "  \$ pytest $A $R $B"
$PY "$A" "$R" "$B"
echo "  ^ test_b errors: fixture 'shared' not found (it is in tests/woo/conftest.py)"

hr
echo "CASE 2  FIX #1 (repo-side): SORT the file list so same-dir files are contiguous"
echo "  \$ pytest $R $A $B          # tests/test_resume sorts before tests/woo/..."
$PY "$R" "$A" "$B"
echo "  ^ contiguous collection of tests/woo -> single Dir node -> all pass"

hr
echo "CASE 3  FIX #2 (upstream-style): same BROKEN order, but baseid-fallback patch loaded"
echo "  \$ PYTHONPATH=. pytest -p fix_matchfactories $A $R $B"
PYTHONPATH=. $PY -p fix_matchfactories "$A" "$R" "$B"
echo "  ^ identity match fails on the duplicate Dir node, baseid match saves it -> all pass"

hr
echo "DIAGNOSTIC  show the two DIFFERENT 'tests/woo' Dir node objects (the root cause)"
cat > diag.py <<'PYEOF'
import _pytest.fixtures as F
_orig = F.FixtureManager.getfixturedefs
def getfixturedefs(self, argname, node):
    r = _orig(self, argname, node)
    if argname == "shared" and node.nodeid.endswith(("test_a", "test_b")):
        fd = (self._arg2fixturedefs.get("shared") or [None])[0]
        woo = {p.nodeid: id(p) for p in node.iter_parents()}.get("tests/woo")
        print(f"    {node.nodeid:60s} match={len(r) if r else 0} "
              f"fixturedef.node id={id(fd.node)} requesting-test's 'tests/woo' id={woo}")
    return r
F.FixtureManager.getfixturedefs = getfixturedefs
PYEOF
echo "  \$ PYTHONPATH=. pytest -p diag $A $R $B"
PYTHONPATH=. python -m pytest -q -p no:cacheprovider -p diag "$A" "$R" "$B" 2>&1 \
    | grep -E "match=" | head
echo "  ^ test_a's 'tests/woo' id == fixturedef.node id  (match=1)"
echo "    test_b's 'tests/woo' id != fixturedef.node id  (match=0) <- duplicate Dir node"
hr

[bluetech]

Hi,

To verify your analysis, can you show the output of pytest --collect-only for the two scenarios (fails and works)?

[RonnyPfannschmidt]

This is the same as the other issue im working on

We accidentally created multiple directory level nodes for the same path

[RonnyPfannschmidt]

closing as duplicate of