StreamingFast Storage Abstraction

dstore is a simple abstraction on top of Local storage and Cloud storage. It handles commonly used functions to store things (locally, or on cloud storage providers), list files, delete, etc..

It is used by StreamingFast.

Features

It currently supports:

• AWS S3 (s3://[bucket]/path?region=us-east-1, with AWS-specific env vars)
  • Minio (through the S3 interface)
• Google Storage (gs://[bucket]/path, with GOOGLE_APPLICATION_CREDENTIALS env var set)
• Azure Blob Storage (az://[account].[container]/path, with AZURE_STORAGE_KEY env var set)
• Local file systems (including virtual of fused-based) (file:/// prefix)

Compression

The compression_config query parameter of a store URL, whatever the scheme, tunes how that store compresses the files it writes:

• zstd stores: <level> or <level>/<window MiB>, for example best, better/32 or best/64. Levels are fastest, default, better and best; the window must be a power of two.
• gzip stores: an integer level from -2 to 9, as defined by compress/gzip: 1 is the fastest, 9 the smallest.

For example, NewDBinStore("gs://bucket/merged-blocks?compression_config=best/32").

On zstd stores, decoder settings can follow the level, separated by commas, or stand alone: best/32,lowmem=false,pool=blocks, pool=cache. They change how the store reads, not what it writes.

• lowmem=false gives each decoder a history buffer of twice the window instead of the window plus 1 MiB. With the default buffer, the decoder copies the whole window back to the start of the buffer about every MiB once an object is larger than its window; with lowmem=false it does so once per window. Objects smaller than window + 1 MiB gain nothing and only use more memory.
• pool=<name> reads with decoders kept in a process-wide pool of that name (1 to 64 letters, digits, - or _), reused across objects and across every store using the same name and lowmem. Pooled decoders decode on the reading goroutine instead of 4 background ones. The pool holds no fixed number of decoders: it keeps those given back by closed readers until garbage collection drops the unused ones. A zstd store naming no pool reads with the default one.
• pool=none gives each object a decoder of its own, with 4 background goroutines, closed with the reader.

Separate pools keep decoders sized for their own window: a decoder grows its buffer for the largest window it has read and keeps it, so a pool shared by 32 MiB and 16 MiB windows ends up with 32 MiB-window buffers in every decoder.

Use commas between settings: Go drops a query value holding a ;.

The store constructor fails when the value is not valid for the store's compression (8 on a zstd store, better on a gzip one, pool=blocks on a gzip one), or when the store has no compression. Sub-stores and clones keep the setting, and NewStore logs it.

Nothing changes on the read side: decoders take the window from the frame header. Every reader needs memory for that window, though, and the zstd command line tool refuses windows above 128 MiB unless run with --long=31 or --memory.

Measured on BNB Chain merged blocks (16-core arm64), relative to the defaults:

config	compressed size	encode	decode
(unset)	100%	879 MB/s	2362 MB/s
better	89.5%	475 MB/s	2466 MB/s
better/32	86.1%	490 MB/s	1347 MB/s
best	86.8%	73 MB/s	2512 MB/s
best/32	77.6%	66 MB/s	1365 MB/s
best/64	73.6%	64 MB/s	892 MB/s
best/128	71.4%	64 MB/s	623 MB/s

Reading 100 MiB objects written with best/32, 10 at a time (16-core arm64):

decoder settings	CPU	peak heap
pool=none	100%	1138 MB
(unset, default pool)	94%	536 MB
lowmem=false,pool=none	38%	2143 MB
lowmem=false (default pool)	28%	844 MB

Testing

The storetests package contains all our integration tests we perform on our store implementation. Some of the store implementations can be tested directly while few others, from Cloud Providers essentially, requires some extra environment variables to run. They are skipped if the correct environment variables for the provider are not set.

Local backends (MinIO + Ceph RGW + fake-gcs-server)

A docker-compose.yml is provided at the root of the repository. It starts:

• MinIO on port 9000 — S3-compatible object storage
• Ceph RGW on port 8080 — built from docker/ceph-local using quay.io/ceph/ceph:v19 (native arm64 + amd64); bootstraps a single-node cluster on first start (~30–60 s)
• fake-gcs-server on port 4443 — GCS-compatible object storage (native arm64 + amd64); data is in-memory (lost on restart)

docker compose up -d

Once the containers are healthy, run the local tests:

STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin" \
STORETESTS_S3_CEPH_STORE_URL="s3://localhost:8080/store-tests?region=none&insecure=true&access_key_id=cephaccesskey&secret_access_key=cephsecretkey" \
STORETESTS_GS_EMULATOR_STORE_URL="gs://store-tests" \
STORAGE_EMULATOR_HOST="localhost:4443" \
go test ./...

Cloud backends

To also run against real cloud providers, supply the relevant environment variables:

STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests" \
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2" \
go test ./...

Note

The bucket names above are placeholders — replace them with real buckets you have access to.

Any variable that is not set will cause the corresponding tests to be skipped automatically.

Contributing

Issues and PR in this repo related strictly to the dstore library.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

License

Apache 2.0