feat: add FileDownloadCrawler

[Mantisus]

Description

• Adds FileDownloadCrawler with support for streaming the response body in the request handler.

Issues

• Closes: Implement a FileDownload crawler #875

Testing

• Added tests for FileDownloadCrawler

[Pijukatel]

For large files, it would be great to have something like https://pypi.org/project/smart-open/ , but our storages lacks interface to support such chunked uploads

[vdusek]

Otherwise it looks great

[vdusek]

From Claude:

The streamed path never sets context.request.loaded_url, whereas the buffered crawl() path does (request.loaded_url = str(response.url)). As a result, in stream mode _check_url_after_redirects is effectively skipped (it's guarded by if loaded_url is not None), so off-domain redirects aren't filtered by the enqueue strategy, and context.request.loaded_url stays None for user code. It's a silent divergence between the two modes. The stream() API doesn't expose the final URL, so this may be hard to fix cleanly, but worth a note/follow-up.

[vdusek]

Claude's nit:

In stream mode navigation_timeout only bounds connection establishment + headers, not the body download (that runs under request_handler_timeout, as the next comment correctly says). This wording reads as if the whole download is bounded by navigation_timeout, so a reader tuning only this value for a large file would still hit the handler timeout.

[vdusek]

Claude's nit:

context.request.url.split('/')[-1] yields '' for a trailing-slash URL, and kvs.set_value(key='') would raise. The runnable example (file_download.py) uses the safer URL(context.request.url).name — worth matching here (and at line 73) so the docstring doesn't teach the more fragile pattern.