Add support for db fixture (test inside transaction) for asyncio tests

[lode-braced]

In using and debugging pytest-django with async tests & fixtures, I ended up hacking together a way to get the db fixture working by enabling the transaction in the sync to async executor thread in which async orm queries run.

This PR aims to integrate that hack as actual functionality into pytest-django.

The async work is done, I also worked to refine the sync side database access: not allowing db access to threads other than the main thread which is running the test (& transaction). That second part is causing some test failures I've yet been unable to fix on 3.9 & django 4.2, will try to fix or drop it from the PR.

edit: I managed to fix the issue with the test failures: my monkeypatches were dropping the self attribute. Ready for review.

Suggestions & feedback welcome.

[kingbuzzman]

This is not a code review

From what I can briefly see; this looks great, i've personally had this issue come up, so happy you've taken this on!

[lode-braced]

@kingbuzzman CI and myself are now happy with where the branch is at, feel free to have a deeper look now.

[kingbuzzman]

@lode-braced can you please sync with main, there are new CI rules

[lode-braced]

@lode-braced can you please sync with main, there are new CI rules

Done!

[kingbuzzman]

5/8 files reviewed. More to follow.

[lode-braced]

@kingbuzzman apologies for the silence, I've been on holiday. The PR should have addressed your remarks where possible, and I've removed the sync code, to be done in another PR once we get this one done.

Can you have another look and close topics as needed?

[lode-braced]

@kingbuzzman Is there anything you need/someone else I should ping to get this reviewed further?

[kingbuzzman]

Hey @lode-braced I haven't been ignoring you deliberately. I've been waiting for others to give some feedback, specially @bluetech

[Artui]

Hey, we're running in some similar issues with the combination of pytest-asyncio, pytest-django and pytest-xdist. Any idea when this PR can be expected to be reviewed/approved? Thanks!

[lode-braced]

Hey, we're running in some similar issues with the combination of pytest-asyncio, pytest-django and pytest-xdist. Any idea when this PR can be expected to be reviewed/approved? Thanks!

To take some pressure off the pytest-django maintainers, I've now extracted the logic from this pr into a separate pytest plugin, that will exist until this pr is merged. Mostly meant for my own convenience, I'm working with quite some async Django projects, but if it helps you too @Artui, feel free to give it a try

see source here
and pypi here

[pcraciunoiu]

@lode-braced thanks for your work on this!

I was able to install that project and opened a PR that fixed some issues I ran into across 2 repos. It's nice being able to remove the autouse fixture I had in place before. Hopefully this is a less hacky solution, seems like it. (although the whole async python/django ecosystem seems quite a mess)

[bluetech]

I rebased this PR (before review). Had to fix some minor conflicts.

[bluetech]

I split the _build_pytest_django_test_case refactor to a separate commit to make the 2nd commit easier to review. I will try to review tomorrow.

[bluetech]

Thanks for the PR @lode-braced. I hoped to merge it for v4.12.0, but yeah, this is a complex topic -- need to get up to speed with asyncio, pytest-asyncio, asgiref, and how Django's own TestCase handles async, before I can evaluate the proposal in this PR. (methinks async was the biggest Python mistake ever, but that's besides the point 😀 )

Here is my understanding so far after a few minutes of looking into it, which might not be entirely accurate yet:

The reason this PR is needed in the first place is that just using

@pytest.mark.asyncio
async def test_it(db):
    await Item.objects.acreate(name="foo")

doesn't rollback the DB changes at the end of the test, i.e. the new Item sticks around.

The reason for that is (I'm mostly guessing here) the DB transaction BEGIN/ROLLBACK that setUpClass executes runs on a different thread than the Item.objects.create. And that's because the Item.objects.acreate doesn't have an AsyncToSync context to latch onto, so the thread_sensitive stuff isn't effective.

What Django's TestCase does is just wrap the test method itself with AsyncToSync, leaving all of the setups and teardowns sync. What this PR does instead (for async tests) is make the entire _django_db_helper async, but using SyncToAsync on the Django setups and teardowns (since they are sync). An advantage of this approach is that pytest fixtures can also be async and share the context.

[bluetech]

If we're going all in on pytest-asyncio, would be better to use pytest_asyncio.is_async_test(request.node) instead.

Although there are several async-support plugins for pytest, e.g. anyio. It would be nice to be agnostic, but maybe not possible.

[lode-braced]

Thanks for the PR @lode-braced. I hoped to merge it for v4.12.0, but yeah, this is a complex topic -- need to get up to speed with asyncio, pytest-asyncio, asgiref, and how Django's own TestCase handles async, before I can evaluate the proposal in this PR. (methinks async was the biggest Python mistake ever, but that's besides the point 😀 )

Here is my understanding so far after a few minutes of looking into it, which might not be entirely accurate yet:

The reason this PR is needed in the first place is that just using

@pytest.mark.asyncio
async def test_it(db):
    await Item.objects.acreate(name="foo")

doesn't rollback the DB changes at the end of the test, i.e. the new Item sticks around.

The reason for that is (I'm mostly guessing here) the DB transaction BEGIN/ROLLBACK that setUpClass executes runs on a different thread than the Item.objects.create. And that's because the Item.objects.acreate doesn't have an AsyncToSync context to latch onto, so the thread_sensitive stuff isn't effective.

What Django's TestCase does is just wrap the test method itself with AsyncToSync, leaving all of the setups and teardowns sync. What this PR does instead (for async tests) is make the entire _django_db_helper async, but using SyncToAsync on the Django setups and teardowns (since they are sync). An advantage of this approach is that pytest fixtures can also be async and share the context.

Hi, thanks for looking at it! I'm happy to see some movement on it.

As a followup on your message:

The root problem is indeed the threads, but I can already save you some time on the reason it runs on different threads, to supplement/correct your guesses: Django's testcase doesn't wrap the method with async_to_sync, as far as I could see at the time of writing the PR, it was setup to only run tests in sync scenarios.

pytest-asyncio is what wraps some work in an async loop, and once in an async loop, the async to sync takes its one thread pool executor which it shares in the loop's context, makes one new sync thread to run al the orm queries in.

In practice, the tests have two threads, running as follows:

T1 (the main thread): run sync fixtures, and async event loops for async fixtures/async tests: the async event loop is started in the main thread and "takes it over" to run the async event loop until complete. Then, Django orm work in the async event loop in T1 spawns a new, shared thread pool executor of size 1, on T2.

Since the django db setup is a sync fixture in pytest django, and that fixture starts the transaction, T1 has a transaction, T2 does not.

Two relevant things:

1. I do think async is quite neat, but do understand it can be a curve. If you'd like to schedule a call where I talk you through the parts that make up the problem here, happy to do so and spread the good word of async ;)
2. I built a better fix in https://github.com/lode-braced/pytest-django-asyncio, where instead of trying to make the transaction setup happen in T2, I made T1 and T2 share the same connection. Reason being that that also allows the use of sync fixtures to use the django orm for an async text, and given the explanation of the two threads above: you know that T1 is busy running the event loop while T2 is able to run requests, so you don't have two threads using the db connection at the same time, which is the thing to avoid due to it being thread safe. I'll port over this new fix into a new PR here too, up to you if you want to go with what is in this PR first or go with what I believe to be the now better option.

[bluetech]

@lode-braced Thanks for the extra details, I will try to understand them soon :)

Django's testcase doesn't wrap the method with async_to_sync, as far as I could see at the time of writing the PR, it was setup to only run tests in sync scenarios.

Just to note on this, Django has been doing this since async support was added:

https://github.com/django/django/blob/ae25a40be07e8a749edf526df37c93e59d4a22c9/django/test/testcases.py#L351-L353

I do think it's meant more as a convenience than a comprehensive solution.

[lode-braced]

https://github.com/django/django/blob/ae25a40be07e8a749edf526df37c93e59d4a22c9/django/test/testcases.py#L351-L353

Oh, TIL! Must have gotten my memory of the test setup code (sync) confused with the actual async test methods being allowed.