`context.request.user_data` mutations are silently dropped on retry

[vdusek]

Description

Mutating context.request.user_data inside a handler doesn't persist across a retry. The change is visible within the same handler run, but silently gone on the next attempt.

Cause: context.request is an isolated deepcopy created for handler execution (RequestHandlerRunResult, src/crawlee/_types.py:277), while the retry reclaims the original request. So in-handler mutations to context.request (including user_data) are dropped on retry. This differs from Crawlee for JS, where request.userData mutations persist across retries.

Surfaced in discussion #2028, where the natural workaround (stash state on request.user_data, read it on the retry) silently fails.

Reproduction

import asyncio

from crawlee.crawlers import BasicCrawler, BasicCrawlingContext


async def main() -> None:
    crawler = BasicCrawler(max_request_retries=3)

    @crawler.router.default_handler
    async def handler(context: BasicCrawlingContext) -> None:
        print(f'retry_count={context.request.retry_count}, flag={context.request.user_data.get("flag")!r}')
        if context.request.retry_count == 0:
            context.request.user_data['flag'] = 'set-on-first-attempt'
            raise RuntimeError('force a retry')

    await crawler.run(['https://example.com/'])


asyncio.run(main())

Actual

retry_count=0, flag=None
retry_count=1, flag=None

Expected

retry_count=0, flag=None
retry_count=1, flag='set-on-first-attempt'

Reproduced on crawlee 1.8.x.

[Mantisus]

This conflicts with the PR - #1585

We need to decide what the final behavior should be.

[Huaian666]

js/python parity is the bigger deal here tbh — silent data loss on retry is brutal to debug when porting handlers from Crawlee JS. deepcopy isolation is fine for handler safety, but maybe an explicit request.persist_user_data() would cover retry-state without changing mutation semantics