iAdd DESIGN doc - dedup - deduplicating backup program Err bitreich.org 70
hgit clone git://bitreich.org/dedup/ git://enlrupgkhuxnvlhsf6lc3fziv5h2hhfrinws65d7roiv6bfj7d652fid.onion/dedup/ URL:git://bitreich.org/dedup/ git://enlrupgkhuxnvlhsf6lc3fziv5h2hhfrinws65d7roiv6bfj7d652fid.onion/dedup/ bitreich.org 70
1Log /scm/dedup/log.gph bitreich.org 70
1Files /scm/dedup/files.gph bitreich.org 70
1Refs /scm/dedup/refs.gph bitreich.org 70
1Tags /scm/dedup/tag bitreich.org 70
1README /scm/dedup/file/README.gph bitreich.org 70
1LICENSE /scm/dedup/file/LICENSE.gph bitreich.org 70
i--- Err bitreich.org 70
1commit a7753b65b2b40ba265e30e8f2f0bda25da7baa53 /scm/dedup/commit/a7753b65b2b40ba265e30e8f2f0bda25da7baa53.gph bitreich.org 70
1parent 42797a877f6efb89a46968af77fe8ab9c0e335fa /scm/dedup/commit/42797a877f6efb89a46968af77fe8ab9c0e335fa.gph bitreich.org 70
hAuthor: sin <sin@2f30.org> URL:mailto:sin@2f30.org bitreich.org 70
iDate: Mon, 6 May 2019 01:00:49 +0100 Err bitreich.org 70
i Err bitreich.org 70
iAdd DESIGN doc Err bitreich.org 70
i Err bitreich.org 70
iDiffstat: Err bitreich.org 70
i A DESIGN | 51 +++++++++++++++++++++++++++++++ Err bitreich.org 70
i Err bitreich.org 70
i1 file changed, 51 insertions(+), 0 deletions(-) Err bitreich.org 70
i--- Err bitreich.org 70
1diff --git a/DESIGN b/DESIGN /scm/dedup/file/DESIGN.gph bitreich.org 70
i@@ -0,0 +1,51 @@ Err bitreich.org 70
i+Design notes Err bitreich.org 70
i+============ Err bitreich.org 70
i+ Err bitreich.org 70
i+There are three main abstractions in the design of dedup: Err bitreich.org 70
i+ Err bitreich.org 70
i+ - The chunker interface Err bitreich.org 70
i+ - The snapshot layer Err bitreich.org 70
i+ - The block layer Err bitreich.org 70
i+ Err bitreich.org 70
i+The block layer Err bitreich.org 70
i+--------------- Err bitreich.org 70
i+ Err bitreich.org 70
i+From the outside world, the block layer is just an abstraction for Err bitreich.org 70
i+dealing with variable length blocks. All blocks are referenced with Err bitreich.org 70
i+their hash. Err bitreich.org 70
i+ Err bitreich.org 70
i+The block layer is arranged into a stack of layers. From top to Err bitreich.org 70
i+bottom these are as follows: Err bitreich.org 70
i+ Err bitreich.org 70
i+ - Generic layer Err bitreich.org 70
i+ - The compression layer Err bitreich.org 70
i+ - The encryption layer Err bitreich.org 70
i+ - The storage layer Err bitreich.org 70
i+ Err bitreich.org 70
i+The generic layer is the one that client code interfaces with. It is Err bitreich.org 70
i+the top level entrypoint to the block layer. Err bitreich.org 70
i+ Err bitreich.org 70
i+The compression layer will prepend a compression descriptor to the Err bitreich.org 70
i+block and then compress the block using snappy or lz4. It is possible Err bitreich.org 70
i+to disable compression in which case a special descriptor is prepended Err bitreich.org 70
i+and the data is passed uncompressed to the layer below. Err bitreich.org 70
i+ Err bitreich.org 70
i+The encryption layer will prepend an encryption descriptor to the Err bitreich.org 70
i+block and then encrypt/authenticate the block using XChaCha20 and Err bitreich.org 70
i+Poly1305. It is possible to disable encryption in which case it acts Err bitreich.org 70
i+as a bypass with a special type of encryption descriptor. Err bitreich.org 70
i+ Err bitreich.org 70
i+The storage layer will prepend a storage descriptor and append the Err bitreich.org 70
i+descriptor and the data to a single backing file. Err bitreich.org 70
i+ Err bitreich.org 70
i+The snapshot layer Err bitreich.org 70
i+------------------ Err bitreich.org 70
i+ Err bitreich.org 70
i+The snapshot abstraction is currently very simplistic. A snapshot is Err bitreich.org 70
i+a file under $repo/archive/<name>. The contents of the file are the Err bitreich.org 70
i+block hashes of the data stored in the snapshot. Err bitreich.org 70
i+ Err bitreich.org 70
i+The chunker interface Err bitreich.org 70
i+--------------------- Err bitreich.org 70
i+ Err bitreich.org 70
i+TBD Err bitreich.org 70
.
Response:
text/plain