Explanation-First
Storage Engine Physics.
Kolmos replaces traditional raw page files with content-addressed explanation segments. Combining FastCDC chunk deduplication, WASM decoders, and Cloudflare R2 object storage.
How Data Moves from Write to Immutable Object
High-frequency incoming records flow through a crash-safe WAL buffer before background compaction synthesizes permanent KSF segments.
Crash-Safe WAL
Incoming INSERTs and UPDATEs append to an fsynced local log with CRC-32 frames. Queries merge WAL overlay state in-memory with sub-millisecond response.
Segment Compaction
Workers group 200,000 rows, evaluating the Minimum Description Length formula across formula miners and prototype medoids to build KSF segments.
FastCDC & CAS Upload
Segments are chunked and hashed via Blake3. Deduplicated blocks upload directly to Cloudflare R2 with atomic conditional-PUT commits.
Storage Engine Specifications
Engineered for extreme data compression, durability, and open perpetual accessibility.
KSF1 Binary Segment Format
Binary SpecificationSegment files utilize LEB128 variable-length integer encoding, aligned columnar chunk offsets, and 64-byte cryptographic segment footers with Blake3 digests.
FastCDC Chunk Deduplication
Content-Addressed StorageContent-Defined Chunking partitions segments into normalized ~64KB chunks (4MB for high-volume S3), deduplicated globally using Blake3 hashes.
WASM Decoder Covenants
Zero Vendor Lock-InEvery segment footer references a sandboxed WASM decoder artifact executing in Wasmtime with strict fuel meters, ensuring data stays decipherable decades into the future.
Two-Tier Query Pruning
Sub-Second Query RoutingHeader Min/Max zone maps skip 90% of segments during range scans; in-memory PK Bloom filter sidecars prune 99% of segments during point lookups.
Zero-Egress Cloud Economics
Cloudflare R2 NativeBuilt ground-up for object storage at $0.015/GB/month with zero egress fees, avoiding traditional cloud database EBS and provisioned IOPS bills.
Crash-Safe WAL Buffer
ACID DurabilityHigh-throughput write-ahead logging with CRC-32 framing buffers row-level mutations and merges query views in real time before background segment compaction.