content-defined-chunking

Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.

DedupBench is a benchmarking tool for content-defined chunking techniques used in data deduplication. It currently supports eleven unique CDC techniques and five different vector instruction sets.

Steadlith reuses unchanged RAG chunks with content-defined identities, cache-aware planning, and transactional indexing.

Fast content-defined chunking in Go.

CDCZip is a deduplication filter meant for use as long range match finder preprocessor for use alongside other compression algorithms.

Find (partial content) duplicate files.

A Rust library for Content-Defined Chunking (CDC).

Print FastCDC rolling hash chunks and checksums.

Synclet is a content-defined, chunk-based file synchronization and backup tool that detects and syncs only the actual changes—efficiently, intelligently, and reliably.

Turn x86-64 hardware into a high-performance POSIX dedup appliance with SMB/FUSE, multi-stage data reduction, and an embedded WebUI.

Ship a search index to a device as signed, content-addressed bundles: fail-closed Ed25519 verification, FastCDC chunking so the next release downloads only the chunks that changed, then FAISS + BM25 rank fusion locally. Python. 309 tests, 98% coverage.

Kernel-safe Rust algorithms for hardware-accelerated byte-stream analysis, content-defined chunking, matching, and runtime dispatch.

Zero-access encryption for Git repositories: Ed25519/X25519 identity (Sequoia OpenPGP), XChaCha20-Poly1305 chunk sealing, keyed content-defined chunking. The auditable core of GitCellar.

Read-only C++20 tool for exact duplicates, hard-linked aliases, and likely file-version families.

Content-defined chunk boundaries so a one paragraph edit re-embeds two chunks, not the document

Open-source toolkit for lossless, reference-based reconstruction of large binary artifacts using deterministic CDC and compact SBD1 seeds.

Clojure implementation of Rabin-Karp rolling hash and chunker

Open-source, self-hostable file sync for people and AI agents, with a Rust daemon and CLI, native macOS Finder integration, scoped sharing, and S3-compatible storage.

ANLA — Agent-Native Lossless Archive. An AI may plan how to pack; a deterministic, model-independent decoder must restore every byte exactly. Two frozen MVP implementations in Python and JavaScript, plus an ANLA 1.0 draft with independent Python and Rust implementations that produce byte-identical archives.

A FUSE-based filesystem implementing content-defined chunking and deduplication in Rust.