v1.0.0-beta · Distributed, Erasure-Coded Engine

Content-Dependent Placement · Measured Results

Measured against a live WarpDrive cluster (symmetric multi-node, gRPC fan-out, Reed-Solomon erasure coding).
Established tools used unmodified against their own data: Lance vector search, DuckDB's tpch extension (official dbgen + queries), and the texmex/SIFT1M-small corpus.

DuckDB · official TPC-H Lance · IVF_PQ vector search SIFT1M-small · published benchmark RS(3,2) erasure coding

Object stores have always treated a stored object as an opaque blob: whatever bytes a client sends get striped, erasure-coded, and handed back whole on read. That's fine for archival and file uploads, but it's a bad fit once an analytical query engine or a vector-index search is the actual client. A Parquet file has column chunks. A Lance index has per-partition structure. A query that only needs one column, or one nprobe's worth of partitions, still has to pay for reconstructing the entire object on a plain erasure-coded store, because the store has no concept of where the part it needs actually lives.

WarpDrive's content-dependent placement lets a bucket opt a workload's own internal byte layout into how it gets striped, so a query can touch only the stripes that actually cover the bytes it needs. The catch is that this can't be a blind optimization: a bad packing decision must never risk availability, so every packer reports its own storage overhead and the system falls back to plain whole-object storage automatically if that overhead crosses a configured threshold. The two results below are the same mechanism applied to two different workloads, an analytical SQL engine and a vector search engine, to check the benefit holds for more than one shape of structure, not just one convenient case.

TPC-H Q6 (selective)
149x
7756.6ms → 51.9ms, row-group pushdown
Lance take latency
up to 41x
20k vectors/768-dim, row-batch packing
SIFT1M-small recall@10
identical
plain vs. packed, every nprobe level
TPC-H Q1 (control)
419.7ms
low selectivity, less to prune

DuckDB's tpch extension generates the official TPC-H dataset (dbgen, sf=1 → 6,001,215 lineitem rows) and exposes the official 22 queries verbatim. Q1 and Q6 are the only two that touch solely lineitem. The exported Parquet file was physically sorted by l_shipdate before upload, a standard clustering technique (the same idea as Databricks' Z-ORDER), which is what makes Q6's one-year date predicate actually prunable via row-group min/max statistics.

Q6 (selective) vs. Q1 (control), median latency

Packed-bucket cost scales down with query selectivity (a stripe boundary to skip past). Plain-bucket cost stays flat regardless of selectivity (no sub-object boundary to exploit at all). Q1 has no plain bar: its near-full-table-scan plain-bucket cost was already evidenced per-request elsewhere and repeating it here is what made an earlier run impractically slow.

Full results

QuerySelectivityBucketMedian latencyNotes
Q6Selective (1-yr window + discount/quantity range)plain7756.6 msno stripe concept, full reconstruction per range-GET
Q6Selectivepacked51.9 ms149x faster, 252 range-GETs, avg 1.0 of 268 stripes touched/request
Q1Low selectivity (~96% of rows match)packed only419.7 mscontrol: 1074 range-GETs, avg 2.2 of 268 stripes/request

Lance's own object_store::aws client pointed at two WarpDrive buckets holding the same dataset, same index, same unmodified query engine. One bucket stored plain (whole-object erasure coding), the other configured to use IvfCentroidPacker, which groups IVF partitions by k-means cluster id and bin-packs within each cluster.

SIFT1M-small · latency (ms)

SIFT1M-small · recall@10

Recall@10 identical between plain and packed at every nprobe level, checked against independently-verifiable ground truth, not just agreement between our own two buckets.

20k vectors/768-dim · row-batch packing latency (ms, log scale)

The dominant cost here was reconstructing the ~60MB data fragment file to materialize 10 result rows. Row-batch packing (20 rows/batch, no cluster metadata needed since row access for a take is uniform-random) gives the largest win measured so far.

SIFT1M-small results

nprobeplain (ms)packed (ms)speeduprecall@10 (plain)recall@10 (packed)
526.356.683.9x0.7430.743
106.081.105.5x0.7650.765
205.991.244.8x0.7680.768

Row-batch packing results (20k vectors/768-dim)

nprobeplain (ms)packed (ms)speedup
4603.514.741x
16604.929.820x
40616.924.026x
Methodology notes and caveats, stated directly:
  • All results use the dev-loop default RS(3,2) (3 data + 2 parity shards), not Fusion's RS(9,6). Re-running at RS(9,6) is the natural next validation step.
  • All measurements are from a local multi-process cluster on one machine (gRPC calls between nodes, disk I/O per node), not yet a multi-VM/cross-datacenter run.
  • No MinIO comparison yet, tracked as follow-up work.
  • SIFT1M-small, not full SIFT1M, the 10k-vector subset, not the full 1M-vector corpus.
  • Every number above came from a script that also independently verifies correctness (byte-identical round-trip, recall@10 against ground truth, or exact result-value equality) in the same run that produced the timing.