Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@
uv add "apify-client[brotli]"
```

[Impit](https://github.com/apify/impit) is the default HTTP client and is installed automatically. To use the
built-in [HTTPX](https://www.python-httpx.org/) client instead, install its optional extra:

```bash
pip install "apify-client[httpx]"
# or
uv add "apify-client[httpx]"
```

- From [conda-forge](https://anaconda.org/conda-forge/apify-client), it can be installed with [conda](https://docs.conda.io/en/latest/):

```bash
Expand Down Expand Up @@ -124,7 +133,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
- **Pluggable HTTP layer** — swap the default [Impit](https://github.com/apify/impit)-based HTTP client for `httpx`, `requests`, `aiohttp`, or any custom implementation ([Custom HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX](https://www.python-httpx.org/) client, or provide any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).
- **Debug logging** — opt-in structured logging on the `apify_client` logger captures request URLs, status codes, retry attempts, and more ([Logging](https://docs.apify.com/api/client/python/docs/concepts/logging)).

Expand Down Expand Up @@ -192,7 +201,7 @@ The full documentation lives at **[docs.apify.com/api/client/python](https://doc
| [Introduction](https://docs.apify.com/api/client/python/docs) | Overview, prerequisites, and a tour of the client. |
| [Quick start](https://docs.apify.com/api/client/python/docs/quick-start) | Authenticate, run an Actor, and fetch its results step by step. |
| [Concepts](https://docs.apify.com/api/client/python/docs/concepts/asyncio-support) | Asyncio, single vs. collection clients, nested clients, error handling, retries, logging, convenience methods, pagination, streaming, custom HTTP clients, timeouts. |
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), use HTTPX as the HTTP client. |
| [Guides](https://docs.apify.com/api/client/python/docs/guides/passing-input-to-actor) | Pass input to an Actor, manage tasks for reusable input, retrieve Actor data, integrate with data libraries (e.g. Pandas), build a custom HTTP client. |
| [Upgrading](https://docs.apify.com/api/client/python/docs/upgrading/upgrading-to-v3) | Migrating between major versions. |
| [API reference](https://docs.apify.com/api/client/python/reference) | Generated reference for every class, method, and model. |
| [Changelog](https://docs.apify.com/api/client/python/docs/changelog) | Release history and breaking changes. |
Expand Down
19 changes: 19 additions & 0 deletions docs/01_introduction/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,25 @@ For better request-body compression, opt in to `brotli`, which compresses better

For details, see [HTTP compression](../02_concepts/13_http_compression.mdx).

The client uses [Impit](https://github.com/apify/impit) as its default HTTP transport. To use the built-in
[HTTPX](https://www.python-httpx.org/) transport, install its optional dependency:

<Tabs>
<TabItem value="PyPI" label="PyPI" default>
```bash
pip install "apify-client[httpx]"
```
</TabItem>
<TabItem value="conda-forge" label="conda-forge">
```bash
conda install conda-forge::apify-client conda-forge::httpx
```
</TabItem>
</Tabs>

See [HTTP clients](../02_concepts/10_custom_http_clients.mdx) for synchronous and asynchronous examples and details
about the shared architecture.

## Quick example

The following example shows how to run an Actor and retrieve its results:
Expand Down
118 changes: 95 additions & 23 deletions docs/02_concepts/10_custom_http_clients.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: custom-http-clients
title: Custom HTTP clients
description: Replace the default HTTP client with a custom implementation.
title: HTTP clients
description: Understand the built-in HTTP clients and the custom client interface.
---

import Tabs from '@theme/Tabs';
Expand All @@ -11,22 +11,26 @@ import ApiLink from '@theme/ApiLink';

import DefaultHttpClientAsyncExample from '!!raw-loader!./code/10_default_http_client_async.py';
import DefaultHttpClientSyncExample from '!!raw-loader!./code/10_default_http_client_sync.py';
import HttpxHttpClientAsyncExample from '!!raw-loader!./code/10_httpx_client_async.py';
import HttpxHttpClientSyncExample from '!!raw-loader!./code/10_httpx_client_sync.py';

import ArchitectureImportsExample from '!!raw-loader!./code/10_architecture_imports.py';

import PluggingInAsyncExample from '!!raw-loader!./code/10_plugging_in_async.py';
import PluggingInSyncExample from '!!raw-loader!./code/10_plugging_in_sync.py';

The Apify API client uses a pluggable HTTP client architecture. By default, it ships with an [Impit](https://github.com/apify/impit)-based HTTP client that handles retries, timeouts, passing headers, and more. You can replace it with your own implementation for use cases like custom logging, proxying, request modification, or integrating with a different HTTP library.
The Apify API client uses a pluggable HTTP layer. It ships with an [Impit](https://github.com/apify/impit)-based default,
offers [HTTPX](https://www.python-httpx.org/) as an optional built-in alternative, and accepts fully custom synchronous
or asynchronous implementations.

## Default HTTP client

When you create an <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> instance, it automatically uses the built-in <ApiLink to="class/ImpitHttpClient">`ImpitHttpClient`</ApiLink> (or <ApiLink to="class/ImpitHttpClientAsync">`ImpitHttpClientAsync`</ApiLink>). This default client provides:

- Automatic retries with exponential backoff for network errors, HTTP 429, and HTTP 5xx responses.
- Configurable timeouts.
- Preparing request data and headers according to the API requirements, including authentication.
- Collecting requests statistics for monitoring and debugging.
- Request compression and preparation of API-compatible data, query parameters, and headers, including authentication.
- API error handling, structured logging, and request statistics.

You can configure the default client through the <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor:

Expand All @@ -43,35 +47,94 @@ You can configure the default client through the <ApiLink to="class/ApifyClient"
</TabItem>
</Tabs>

## Built-in HTTPX client

The package also provides <ApiLink to="class/HttpxHttpClient">`HttpxHttpClient`</ApiLink> and
<ApiLink to="class/HttpxHttpClientAsync">`HttpxHttpClientAsync`</ApiLink>. They use the same request preparation,
compression, retry policy, timeout tiers and growth, error handling, logging, and statistics as the default Impit clients, with
[HTTPX](https://www.python-httpx.org/) as the transport.

HTTPX is an optional dependency. Install `apify-client[httpx]`, then pass the appropriate client to
<ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink>. Impit remains
the default even when the HTTPX extra is installed.

```bash
pip install "apify-client[httpx]"
# or
uv add "apify-client[httpx]"
```

<Tabs>
<TabItem value="HttpxAsyncExample" label="Async client" default>
<CodeBlock className="language-python">
{HttpxHttpClientAsyncExample}
</CodeBlock>
</TabItem>
<TabItem value="HttpxSyncExample" label="Sync client">
<CodeBlock className="language-python">
{HttpxHttpClientSyncExample}
</CodeBlock>
</TabItem>
</Tabs>

Configure retries, timeout tiers, default headers, and compression on the HTTPX client instance. The token passed to
`with_custom_http_client` is applied automatically unless the HTTP client already has an `Authorization` header.
The examples use the clients as context managers so their connection pools are closed deterministically. If a context
manager does not fit your application's lifecycle, call `close()` on `HttpxHttpClient` or `await aclose()` on
`HttpxHttpClientAsync` during shutdown.

Timeout values are passed to the selected transport. Impit treats them as whole-request timeouts, while HTTPX applies
its connect, read, write, and pool timeout semantics. In particular, an HTTPX read timeout limits inactivity between
chunks rather than the total duration of a streamed response. The `no_timeout` option disables HTTPX's timeouts.

## Architecture

The HTTP client system is built on two key abstractions:
Internally, the HTTP client hierarchy has three layers:

- A common internal base contains configuration and utilities shared by synchronous and asynchronous clients, including
headers, request-body preparation, parameters, compression, and timeout tiers. It is not a public extension point.
- <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> and <ApiLink to="class/HttpClientAsync">`HttpClientAsync`</ApiLink>
add the synchronous or asynchronous request pipeline, retry loop, transport hooks, and lifecycle interface.
- The built-in Impit and HTTPX classes inherit directly from the corresponding sync or async class and adapt the
underlying transport.

`HttpClient.is_timeout_error(exc)` and `HttpClientAsync.is_timeout_error(exc)` provide the public, transport-neutral way
to determine whether an exception is a timeout. Their shared implementation recognizes Python's `TimeoutError`;
transport adapters override it when their HTTP library defines additional timeout exception types. This lets
higher-level features such as streamed logs classify timeouts without depending on Impit, HTTPX, or private
implementation details.

Responses use one separate abstraction:

- <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> / <ApiLink to="class/HttpClientAsync">`HttpClientAsync`</ApiLink> - Abstract base classes that define the interface. Extend one of these to create a custom HTTP client by implementing the `call` method.
- <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> - A [runtime-checkable protocol](https://docs.python.org/3/library/typing.html#typing.runtime_checkable) that defines the expected response shape. Any object with the required attributes and methods satisfies the protocol — no inheritance needed.

To plug in your custom implementation, use the <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink> class method.

All of these are available as top-level imports from the `apify_client` package:
The built-in Impit and HTTPX classes are thin transport adapters over the request implementation in `HttpClient` and
`HttpClientAsync`. Custom transport adapters implement the request, error-classification, and lifecycle hooks. They
inherit request construction, retries, timeout growth, API error conversion, logging, and statistics from the base.

All of these are available from the `apify_client.http_clients` module:

<CodeBlock className="language-python">
{ArchitectureImportsExample}
</CodeBlock>

### The call method
### The transport contract

The `call` method receives all the information needed to make an HTTP request:
The public `call` method provides the shared request pipeline. A concrete transport implements these hooks:

- `method` - HTTP method (`GET`, `POST`, `PUT`, `DELETE`, etc.).
- `url` - Full URL to make the request to.
- `headers` - Additional headers to include.
- `params` - Query parameters to append to the URL.
- `data` - Raw request body (mutually exclusive with `json`).
- `json` - JSON-serializable request body (mutually exclusive with `data`).
- `stream` - Whether to stream the response body.
- `timeout` - Timeout for the request as a `timedelta`.
- `send_request(...)` sends one prepared request and returns an `HttpResponse`. The inherited `call` needs it, so
every transport adapter has to implement it.
- `is_retryable_transport_error(exc)` classifies transport failures for the shared retry loop. The default classifies
nothing as retryable, so a transport that skips it gives up on the first connection failure.
- `is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The
default recognizes Python's `TimeoutError`.
- `close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a
transport that owns no pool or session.

It must return an object satisfying the <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> protocol.
The `@override` decorators in the built-in Impit and HTTPX adapters make these implementations explicit and allow type
checkers to catch misspelled or incompatible overrides.

### The HTTP response protocol

Expand All @@ -93,6 +156,10 @@ It must return an object satisfying the <ApiLink to="class/HttpResponse">`HttpRe

:::note
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX](https://www.python-httpx.org/) already satisfy this protocol out of the box.

For a streamed response, consume the body inside its context manager with `iter_bytes()` / `aiter_bytes()`, or call
`read()` / `aread()` before accessing `content`. Some transports, including HTTPX, intentionally reject `content` on
an unread streamed response.
:::

### Plugging it in
Expand All @@ -115,18 +182,23 @@ Use the <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.wit
After that, all API calls made through the client will go through your custom HTTP client.

:::warning
When using a custom HTTP client, you are responsible for constructing the request, handling retries, timeouts, and errors yourself. The default retry logic is not applied.
If you override `call` itself, your implementation becomes responsible for request preparation, retries, timeouts, API
error conversion, logging, and statistics. Implementing the transport hooks and inheriting `call` keeps the shared
behavior.
:::

## Use cases

Custom HTTP clients might be useful when you need to:
Custom HTTP clients might be useful when the built-in Impit and HTTPX clients do not cover your requirements, for
example when you need to:

- **Use a different HTTP library** - Swap Impit for [httpx](https://www.python-httpx.org/), [requests](https://requests.readthedocs.io/), or [aiohttp](https://docs.aiohttp.org/).
- **Use a different HTTP library** - Integrate [requests](https://requests.readthedocs.io/), [aiohttp](https://docs.aiohttp.org/), or another transport.
- **Route through a proxy** - Add proxy support or request routing.
- **Implement custom retry logic** - Use different backoff strategies or retry conditions.
- **Log requests and responses** - Track API calls for debugging or auditing.
- **Modify requests** - Add custom fields, modify the body, or change headers.
- **Collect custom metrics** - Measure request latency, track error rates, or count API calls.

For a step-by-step walkthrough of building a custom HTTP client, see the [Using HTTPX as the HTTP client](/api/client/python/docs/guides/custom-http-client-httpx) guide.
For complete synchronous and asynchronous implementations over a transport with a different response API, see
[Build a custom HTTP client](../03_guides/05_custom_http_client.mdx). You can also refer to the
<ApiLink to="class/HttpClient">`HttpClient` API reference</ApiLink> for the synchronous contract.
17 changes: 17 additions & 0 deletions docs/02_concepts/code/10_httpx_client_async.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import asyncio

from apify_client import ApifyClientAsync
from apify_client.http_clients import HttpxHttpClientAsync


async def main() -> None:
async with HttpxHttpClientAsync() as http_client:
client = ApifyClientAsync.with_custom_http_client(
token='MY-APIFY-TOKEN',
http_client=http_client,
)
print(await client.actor('apify/hello-world').get())


if __name__ == '__main__':
asyncio.run(main())
11 changes: 11 additions & 0 deletions docs/02_concepts/code/10_httpx_client_sync.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from apify_client import ApifyClient
from apify_client.http_clients import HttpxHttpClient


def main() -> None:
with HttpxHttpClient() as http_client:
client = ApifyClient.with_custom_http_client(
token='MY-APIFY-TOKEN',
http_client=http_client,
)
print(client.actor('apify/hello-world').get())
27 changes: 17 additions & 10 deletions docs/02_concepts/code/10_plugging_in_async.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,34 @@
from typing import Any
from typing_extensions import override

from apify_client import ApifyClientAsync
from apify_client.http_clients import HttpClientAsync, HttpResponse
from apify_client.types import Timeout

TOKEN = 'MY-APIFY-TOKEN'


class MyHttpClientAsync(HttpClientAsync):
"""Custom async HTTP client."""

async def call(
@override
async def send_request(
self,
*,
method: str,
url: str,
headers: dict[str, str] | None = None,
params: dict[str, Any] | None = None,
data: str | bytes | bytearray | None = None,
json: Any = None,
stream: bool | None = None,
timeout: Timeout = 'medium',
) -> HttpResponse: ...
headers: dict[str, str],
content: bytes | None,
timeout: float | None,
stream: bool,
) -> HttpResponse:
"""Send one request through the custom transport."""
raise NotImplementedError

@override
def is_retryable_transport_error(self, exc: Exception) -> bool:
# List the transport's transient failures here, e.g. its timeout
# and connection errors. Returning False for everything opts out
# of transport retries entirely.
return isinstance(exc, TimeoutError)


async def main() -> None:
Expand Down
27 changes: 17 additions & 10 deletions docs/02_concepts/code/10_plugging_in_sync.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,34 @@
from typing import Any
from typing_extensions import override

from apify_client import ApifyClient
from apify_client.http_clients import HttpClient, HttpResponse
from apify_client.types import Timeout

TOKEN = 'MY-APIFY-TOKEN'


class MyHttpClient(HttpClient):
"""Custom sync HTTP client."""

def call(
@override
def send_request(
self,
*,
method: str,
url: str,
headers: dict[str, str] | None = None,
params: dict[str, Any] | None = None,
data: str | bytes | bytearray | None = None,
json: Any = None,
stream: bool | None = None,
timeout: Timeout = 'medium',
) -> HttpResponse: ...
headers: dict[str, str],
content: bytes | None,
timeout: float | None,
stream: bool,
) -> HttpResponse:
"""Send one request through the custom transport."""
raise NotImplementedError

@override
def is_retryable_transport_error(self, exc: Exception) -> bool:
# List the transport's transient failures here, e.g. its timeout
# and connection errors. Returning False for everything opts out
# of transport retries entirely.
return isinstance(exc, TimeoutError)


def main() -> None:
Expand Down
Loading
Loading