Colby Serpa on Nostr: We spent a long-time crafting a standard for file chunking by evolving Merkle DAGs, ...

We spent a long-time crafting a standard for file chunking by evolving Merkle DAGs, which are modified Merkle Trees used for file storage.

These Scionic Merkle DAGs are a significant improvement over IPFS Merkle DAGs. https://github.com/HORNET-Storage/scionic-merkletree

IPFS is arguably one of the most popular P2P file sharing platforms in the world, as you know. Their entire system is cryptographically secure due to their Merkle DAGs.

Our merkle branches contain fewer useless hashes. As the folder grows, the branch size decreases logarithmically compared to Merkle DAGs. Stats can be seen on the GitHub page.

These new Scionic Merkle DAGs can be used for a user-sever model where relays store files beyond notes, or a P2P model as you described here. We’re setting up a #Nostr relay that supports them now.

I don’t mind pursuing both futures, but I lean toward the paid user-server model. I’d be curious to see where you and JB take the P2P model, as Robin Linus and I are working on a way to paid relays over Lightning in an atomic way, with preimages as file chunks.

quoting note12fx…trg9

Our predecessor, the Merkle DAG, is depicted by the white graph while the dark-mode graph showcases the new Scionic Merkle Tree.

In Merkle DAGs, each DAG leaf folder houses a list of children (Child1, Child2, etc). However, Scionic Merkle Trees transform this list into a classic Merkle tree. By doing so, a tiny classic Merkle tree is created inside every DAG leaf folder.

Users can now use a classic Merkle branch to acquire a file from the DAG without downloading the hash of every other file within the same folder as the requested file, making the process quicker and less costly. note14vd…hxa3