A high-performance, in-memory streaming data engine built on Log-Magnitude Modular Partitioning and Query-Driven Mathematical Bounding. Engineered in C++ and exposed via Cython, this architecture bridges the classic database trade-off between ingest-optimized hash tables and query-optimized tree structures.
- Blazing Ingestion Velocity: Sustains >350,000 inserts/sec with zero tree rebalancing, locking, or background compaction overhead.
-
True
$O(1)$ Point Lookups: Direct arithmetic routing via(factor, quotient, remainder)tuples into nested C++ STL hash maps. - Query-Driven Range Retrieval ($O(K)$): Replaces physical tree traversal with closed-form mathematical bounding, achieving 13.45x to 15.80x speedups over standard Python linear scans.
-
Zero-Copy Memory Access: Utilizes C++
constreferences (const std::vector<int64_t>&) to eliminate redundant heap allocations during query execution.
Instead of forcing incoming records into a globally sorted tree, each integer
The value is then decomposed into a quotient and remainder:
Data is stored in a cache-friendly nested C++ container:
std::unordered_map<int64_t, std::unordered_map<int64_t, std::vector<int64_t>>> d_sort;When executing a range query q_start to q_end) that mathematically intersect with
Benchmarks executed on 500,000 random integers (domain up to
| Method | Avg. Query Latency | Speedup vs. Baseline |
|---|---|---|
| Standard Python List Scan | 42.45 ms |
1.00x (Baseline) |
C++ Modular Engine (range_search) |
3.16 ms |
13.45x faster |
C++ Modular Engine (range_search_unordered) |
2.69 ms |
15.80x faster |
- Ingestion Throughput: 500,000 records inserted in
~0.42s(~1.18M ops/sec peak C++ loop / >350k ops/sec end-to-end Python stream).
-
Python 3.8+
-
C++ compiler supporting C++11 or later (
g++,clang, or MSVC)Run below command ion terminal
sudo apt update && sudo apt install build-essentialg++ --versionRun below command in powershell
winget install --id=MartinStorsjo.LLVM-MinGW.UCRT -e(Download Visual Studio Build Tools for C++)[https://visualstudio.microsoft.com/visual-cpp-build-tools/]
Post that, install file: vs_BuildTools.exe
After installation completed, from "Visual Studio Build Tool" manager, tick 'Desktop development with C++' and then from Optional, tick 'MSVC Build Tools for x64/x86 and Windows 10/11 SDK' and press install button.
(Optional) Screen may ask to restart your system.
Restart powershell and run below command in powershell
clang++ --version -
Cython
- Clone the repository:
git clone https://github.com/CodersALL/modular-streaming-engine.git cd modular-streaming-engine - Install dependencies:
pip install cython setuptools # OR pip install -r requirements.txt - Build the Cython C++ extension in-place:
python setup.py build_ext --inplace
import bucket_search
# 1. Initialize the C++ Streaming Engine
engine = bucket_search.StreamingModularEngine()
# 2. Stream data in O(1) amortized time
for num in [5, 12, 18, 25, 100, 200, 1001, 1010, 1999]:
engine.insert(num)
# 3. Point Lookup in O(1) time
exists = engine.search(1010) # True
# 4. Query-Driven Range Search (Sorted)
sorted_results = engine.range_search(20, 1020)
print("Sorted matches:", sorted_results)
# 5. Query-Driven Range Search (Unordered - Maximum Throughput)
raw_results = engine.range_search_unordered(20, 1020)
print("Unordered matches:", raw_results)├── bucket_search.pyx # Core Cython/C++ modular engine implementation
├── setup.py # Cython build configuration with -O3 compiler flags
├── benchmark.py # Performance comparison suite vs. Python baselines
└── README.md # Project documentation
- Log-magnitude modular partitioning in C++ STL
- Query-driven mathematical bounding (
range_search) - Zero-copy reference optimization & unordered retrieval (
range_search_unordered) - Binary disk serialization & persistence layer
- Multi-threaded lock-free ingestion pipeline
- Support for negative integers and floating-point quantization
If you use this architecture or reference the findings in your research, please cite:
@article{modular_streaming_engine_2026,
title={Bypassing the Hash-Tree Trade-Off: A Query-Driven Modular Architecture for High-Throughput Streaming and Range Retrieval},
author={SOUMYA GHOSH},
year={2026},
journal={}
}This project is licensed under the MIT License - see the LICENSE file for details.