Object Storage · S3-Compatible · Rust

WarpDrive

Object storage built for disaggregated systems

Warpdrive is a purpose-built S3-compatible object store focused on workloads that demand high throughput — storage-disaggregated databases, ML training pipelines, and agentic workflows that move computation close to data.

An S3ish rocket store

Modern databases, agents, and training systems are building on top of object stores. But historically, high-efficiency systems have never benefited from an opaque storage layer. Object stores were designed for archival and customer file uploads, and that opacity was fine for those use cases. Now latency-sensitive systems are building on the same interface. Through our characterization work we have identified a few early parameters that capture how disaggregated workloads actually use storage, and we expose these so developers can tune them analytically from their own workload behavior. We think both sides benefit when the storage layer is legible and you can reason about its internals, develop against its behavior, and work together toward a system that is genuinely cost-efficient.

Disaggregated databases

Modern databases like Neon, SlateDB, and TidesDB separate their storage layer from compute. They ship their WAL segments, SSTables, and layer files to object storage — and expect it to keep up.

Agentic workloads

Agents manage state, memory, and tool outputs at runtime. They need fast, consistent object storage with predictable latency — not the eventual consistency and cold-start penalties of cloud S3.

Training & checkpointing

ML training jobs checkpoint model state to object storage continuously. Warpdrive provides the throughput and S3 compatibility to serve as a local or rack-local checkpoint store without changing a line of training code.

What Warpdrive provides

A storage layer designed for the systems that need it most.

S3-compatible API

Drop-in replacement for AWS S3. Any client or library that speaks S3 works without modification — boto3, aws-sdk-rust, the official CLIs.

High throughput

Built for workloads where throughput and low latency matter: serverless databases, ML checkpointing, and distributed system state storage.

Compute-storage separation

First-class support for storage-disaggregated architectures. Run storage and compute independently, scale them separately, attach either to S3 clients.

Installation

Warpdrive ships as a single native binary. Choose your install method below.

amd64 / x86_64 arm64 — coming soon macOS — coming soon Windows — coming soon
# Add the Warpdrive apt repository
curl -fsSL https://releases.warpdrive.dev/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/warpdrive.gpg

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/warpdrive.gpg] \
  https://releases.warpdrive.dev/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/warpdrive.list

sudo apt-get update && sudo apt-get install -y warpdrive
Requires Ubuntu 20.04+ or Debian 11+. The binary is warp_drive after install.
Rust 1.75+ Linux macOS Windows — coming soon
# Clone and build
git clone https://github.com/vitality-ai/warpdrive
cd warpdrive/server
cargo build --release

# Run (must be executed from warpdrive/server/)
WARPDRIVE_ADMIN_ACCESS_KEY=mykey \
WARPDRIVE_ADMIN_SECRET_KEY=mysecret \
  ./target/release/warp_drive
The server needs a writable storage/ and logs/ directory in the working directory. It listens on port 9710 by default.
amd64 arm64
# Run with Docker
docker run -d \
  -p 9710:9710 \
  -e WARPDRIVE_ADMIN_ACCESS_KEY=mykey \
  -e WARPDRIVE_ADMIN_SECRET_KEY=mysecret \
  -v $(pwd)/data:/data \
  ghcr.io/vitality-ai/warpdrive:latest
A docker-compose.yml is available in the repo for local development.

Quick smoke-test with boto3 after the server is running:

pip install boto3

python3 - <<EOF
import boto3
s3 = boto3.client('s3', endpoint_url='http://localhost:9710',
    aws_access_key_id='mykey', aws_secret_access_key='mysecret',
    region_name='us-east-1',
    config=boto3.session.Config(s3={'addressing_style': 'path'}))
s3.create_bucket(Bucket='test')
s3.put_object(Bucket='test', Key='hello', Body=b'world')
print(s3.get_object(Bucket='test', Key='hello')['Body'].read())
EOF

Verified integrations

Tested against real production systems using their native S3 clients — no shims, no compatibility layers.

In pipeline: LangGraph · LlamaIndex · Ray · PyTorch Lightning

Workloads

Measured against real engines running their own real workloads, OLTP and OLAP alike, not synthetic microbenchmarks.

OLTP · Neon, serverless Postgres

TPS at 16 tenants, cold start
10xwith co-located slabs
4.1 → 41.8 TPS
Cold-start GETs at 16 tenants
11xfewer
2,981 → 231 GETs
Full OLTP results with charts →

OLAP · DuckDB TPC-H & Lance vector search

TPC-H Q6 (selective)
149xfaster
7756.6ms → 51.9ms
Lance take latency
up to 41xfaster
row-batch packing
SIFT1M-small recall@10
identicalplain vs packed
every nprobe level
Full OLAP results with charts →

Distributed engine, v1.0.0-beta

WarpDrive also runs as a multi-node, erasure-coded cluster with pluggable content-dependent placement: a bucket can opt a workload's own structure (Parquet column chunks, vector-index partitions) into how its bytes are striped and erasure-coded. Measured against established tools, not synthetic benchmarks: DuckDB's official TPC-H (via its own dbgen) and Lance's IVF_PQ vector search.

Finding: content-dependent placement's benefit scales with query selectivity. A selective TPC-H query (Q6, pruned to the row groups it actually needs) goes from 7756.6ms to 51.9ms, a 149x speedup, while a low-selectivity query (Q1, ~96% of rows match) only improves modestly, since there's less to prune either way. On Lance vector search, recall@10 stays bit-for-bit identical between plain and packed buckets at every nprobe level, placement changes where bytes live, never which bytes come back.
TPC-H Q6 · selective
149xfaster
7756.6ms → 51.9ms · row-group pushdown
Lance take latency
41xfaster
20k vectors/768-dim · row-batch packing
SIFT1M-small recall@10
0delta
identical, plain vs. packed, every nprobe
TPC-H Q1 · control
419.7ms
low selectivity, packed only
Full distributed engine report with charts →
vs MinIO, 5-node cluster
3.0xthroughput
vs MinIO, single node
1.5-1.7xthroughput
Write durability
fsyncgroup-commit
Durability & MinIO comparison report →

Research

WarpDrive's design work feeds back into peer-reviewed research, not just internal benchmarking.

VLDB 2026 · CMDS Workshop

WarpDrive: A Composable Storage Substrate for Disaggregated Data Systems — accepted as a lightning talk at CMDS (Composable Data Management Systems), the VLDB 2026 workshop. Read the position paper →

HiPC · submitted, under review

WarpDial — a case for reconfigurable placement policies and explainability in storage SLAs. Currently under review; we'll link the paper once a decision is in.