PebbleFS
Go · SQLite · SHA-256
An open-source content-addressable storage engine that unifies USB drives, HDDs, SSDs, SD cards, and other commodity storage into a fault-tolerant distributed storage pool.
Demo
Key Highlights & Core Contributions
- Multi-Disk Storage Pooling: Aggregated heterogeneous storage devices (USB drives, HDDs, SSDs, flash drives) into a single logical pool with persistent disk registration.
- Content-Addressed Chunks: Files are chunked and addressed by SHA-256, giving automatic deduplication and deterministic placement via directory sharding.
- Service-Oriented Architecture: Clean separation between the CLI, service layer, SQLite-backed metadata store, storage manager, and chunk engine.
- Full File Lifecycle: Upload, download, listing, info, deletion, and storage statistics behind a small CLI (
disk add,put,get,list,info,rm).
Storage Layout
Each registered disk contains a .pebble/ directory with a disk.json manifest and sharded chunks/ tree, so data stays portable and inspectable per device.
Current Status & Roadmap
Currently supports single-node operation with multiple devices, persistent metadata, and deduplicated chunk storage. Planned next: verification, crash recovery, WAL mode, garbage collection, replication, and eventually distributed clustering with a FUSE filesystem interface.