{"id":1773,"date":"2023-04-18T10:51:17","date_gmt":"2023-04-18T08:51:17","guid":{"rendered":"https:\/\/perso.uclouvain.be\/tom.barbette\/?p=1773"},"modified":"2023-04-18T12:15:27","modified_gmt":"2023-04-18T10:15:27","slug":"looking-for-the-right-archive-format-with-deduplication","status":"publish","type":"post","link":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/","title":{"rendered":"Looking for the right archive format with deduplication"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I&#8217;m often copying my files to an external disk as a backup option. So far, nothing fancy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, I end up with many copies of more or less the same folders over time.  And that takes a lot of space for nothing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Incremental backups, that only add new files to an archive, might look as a solution for that, but over time the paths changes, the structure changes. So the deduplication would break.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One solution could be to format my disk with a system such as ZFS or BTRFS which have a deduplication built-in. However I can&#8217;t use my disk for quick operations\/archiving on Windows, nor on Mac (easily at least).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So I want an archive format with:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Deduplication<\/li>\n\n\n\n<li>Some compression (lot of source code files)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">One might think, and I often came across that statement while searching on the web for a solution, that one shouldn&#8217;t worry about that as compression algorithm inheritely support deduplication. That is actually wrong, they&#8217;re not made to merge very large patterns. The following experiment shows it with file of random data (incompressible) :<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">-rwxrwxrwx 1 tbarbette tbarbette 10M Apr 17 21:09 binA<br>-rwxrwxrwx 1 tbarbette tbarbette 10M Apr 17 21:10 binACOPY<br>-rwxrwxrwx 1 tbarbette tbarbette 10M Apr 17 21:09 binB<br>-rwxrwxrwx 1 tbarbette tbarbette 256M Apr 17 21:19 binBIG<br>-rwxrwxrwx 1 tbarbette tbarbette 256M Apr 17 21:19 binBIGCOPY<br>-rwxrwxrwx 1 tbarbette tbarbette 10M Apr 17 21:09 binC<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s the size for a few formats :<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><\/td><td>Size<\/td><td>Time<\/td><td>Comma,d<\/td><\/tr><tr><td>tar.gz<\/td><td>553M<\/td><td>19.48s<\/td><td>tar -cpazf [archive] [file]<\/td><\/tr><tr><td>tar.xz<\/td><td>553M<\/td><td>303s<\/td><td>tar -cpaJf [archive] [file]<\/td><\/tr><tr><td>zpaq<\/td><td>287M<\/td><td>18.7s<\/td><td>zpaq add [archive] [file]<\/td><\/tr><tr><td>wim<\/td><td>287M<\/td><td>4.4s<\/td><td>7zz a [archive].wim [file]<\/td><\/tr><tr><td>.7z<\/td><td>543M<\/td><td>37s<\/td><td>7zz a [archive].7z [file]<\/td><\/tr><tr><td>.7z (512M dictionnary)<\/td><td>287M<\/td><td>140s<\/td><td>7zz a [archive].7z -m0=LZMA2:d512m [file]<\/td><\/tr><tr><td>.7z (1513M dictionnary)<\/td><td>287M<\/td><td>114s<\/td><td>7zz a [archive].7z -m0=LZMA2:d1513m [file]<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">zpaq is the clear winner considering it can also compress at the same time, however it&#8217;s not very widespread, and has no good GUI available. I&#8217;m wondering about recovery in case of problems.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">wim has no compression, so it will need to be encapsulated in something else. The problem is then to add some files. I have to uncompress the inner wim format first. The idea being to save the same computer again and again, one of those archives is 200G, so it means adding a single file would take a huge time. While zpaq can add one quite fast.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">7zip with a dictionary large enough to hold big duplicates would seem to be a good compromise. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Any input?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m often copying my files to an external disk as a backup option. So far, nothing fancy. However, I end up with many copies of more or less the same folders over time. And that takes a lot of space for nothing. Incremental backups, that only add new files to an archive, might look as &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Looking for the right archive format with deduplication&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[64,25,16],"tags":[],"class_list":["post-1773","post","type-post","status-publish","format-standard","hentry","category-misc","category-server-management","category-unix"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Looking for the right archive format with deduplication - Tom Barbette<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Looking for the right archive format with deduplication - Tom Barbette\" \/>\n<meta property=\"og:description\" content=\"I&#8217;m often copying my files to an external disk as a backup option. So far, nothing fancy. However, I end up with many copies of more or less the same folders over time. And that takes a lot of space for nothing. Incremental backups, that only add new files to an archive, might look as &hellip; Continue reading &quot;Looking for the right archive format with deduplication&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/\" \/>\n<meta property=\"og:site_name\" content=\"Tom Barbette\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-18T08:51:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-04-18T10:15:27+00:00\" \/>\n<meta name=\"author\" content=\"Tom Barbette\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@TomBarbette\" \/>\n<meta name=\"twitter:site\" content=\"@TomBarbette\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tom Barbette\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/\"},\"author\":{\"name\":\"Tom Barbette\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#\\\/schema\\\/person\\\/7f791766092e1207bc7a94a65e74f4fd\"},\"headline\":\"Looking for the right archive format with deduplication\",\"datePublished\":\"2023-04-18T08:51:17+00:00\",\"dateModified\":\"2023-04-18T10:15:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/\"},\"wordCount\":423,\"publisher\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#organization\"},\"articleSection\":[\"Misc\",\"Server management\",\"Unix\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/\",\"url\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/\",\"name\":\"Looking for the right archive format with deduplication - Tom Barbette\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#website\"},\"datePublished\":\"2023-04-18T08:51:17+00:00\",\"dateModified\":\"2023-04-18T10:15:27+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/looking-for-the-right-archive-format-with-deduplication\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Looking for the right archive format with deduplication\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#website\",\"url\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/\",\"name\":\"Tom Barbette\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#organization\",\"name\":\"Efficiency of Networked Systems Group\",\"url\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/cropped-logo-uclouvain-2021-barbette.png\",\"contentUrl\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/wp-content\\\/uploads\\\/2022\\\/04\\\/cropped-logo-uclouvain-2021-barbette.png\",\"width\":512,\"height\":512,\"caption\":\"Efficiency of Networked Systems Group\"},\"image\":{\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/TomBarbette\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/tom-barbette\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/#\\\/schema\\\/person\\\/7f791766092e1207bc7a94a65e74f4fd\",\"name\":\"Tom Barbette\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g\",\"caption\":\"Tom Barbette\"},\"sameAs\":[\"https:\\\/\\\/www.tombarbette.be\"],\"url\":\"https:\\\/\\\/perso.uclouvain.be\\\/tom.barbette\\\/author\\\/tom\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Looking for the right archive format with deduplication - Tom Barbette","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/","og_locale":"en_US","og_type":"article","og_title":"Looking for the right archive format with deduplication - Tom Barbette","og_description":"I&#8217;m often copying my files to an external disk as a backup option. So far, nothing fancy. However, I end up with many copies of more or less the same folders over time. And that takes a lot of space for nothing. Incremental backups, that only add new files to an archive, might look as &hellip; Continue reading \"Looking for the right archive format with deduplication\"","og_url":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/","og_site_name":"Tom Barbette","article_published_time":"2023-04-18T08:51:17+00:00","article_modified_time":"2023-04-18T10:15:27+00:00","author":"Tom Barbette","twitter_card":"summary_large_image","twitter_creator":"@TomBarbette","twitter_site":"@TomBarbette","twitter_misc":{"Written by":"Tom Barbette","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/#article","isPartOf":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/"},"author":{"name":"Tom Barbette","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#\/schema\/person\/7f791766092e1207bc7a94a65e74f4fd"},"headline":"Looking for the right archive format with deduplication","datePublished":"2023-04-18T08:51:17+00:00","dateModified":"2023-04-18T10:15:27+00:00","mainEntityOfPage":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/"},"wordCount":423,"publisher":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#organization"},"articleSection":["Misc","Server management","Unix"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/","url":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/","name":"Looking for the right archive format with deduplication - Tom Barbette","isPartOf":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#website"},"datePublished":"2023-04-18T08:51:17+00:00","dateModified":"2023-04-18T10:15:27+00:00","breadcrumb":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/looking-for-the-right-archive-format-with-deduplication\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/perso.uclouvain.be\/tom.barbette\/"},{"@type":"ListItem","position":2,"name":"Looking for the right archive format with deduplication"}]},{"@type":"WebSite","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#website","url":"https:\/\/perso.uclouvain.be\/tom.barbette\/","name":"Tom Barbette","description":"","publisher":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/perso.uclouvain.be\/tom.barbette\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#organization","name":"Efficiency of Networked Systems Group","url":"https:\/\/perso.uclouvain.be\/tom.barbette\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#\/schema\/logo\/image\/","url":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-content\/uploads\/2022\/04\/cropped-logo-uclouvain-2021-barbette.png","contentUrl":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-content\/uploads\/2022\/04\/cropped-logo-uclouvain-2021-barbette.png","width":512,"height":512,"caption":"Efficiency of Networked Systems Group"},"image":{"@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/TomBarbette","https:\/\/www.linkedin.com\/in\/tom-barbette"]},{"@type":"Person","@id":"https:\/\/perso.uclouvain.be\/tom.barbette\/#\/schema\/person\/7f791766092e1207bc7a94a65e74f4fd","name":"Tom Barbette","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/aecd99cba3b4d91d3775fde3219bfa362cd99acfbe95848ec303b9685f366fcd?s=96&r=g","caption":"Tom Barbette"},"sameAs":["https:\/\/www.tombarbette.be"],"url":"https:\/\/perso.uclouvain.be\/tom.barbette\/author\/tom\/"}]}},"_links":{"self":[{"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/posts\/1773","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/comments?post=1773"}],"version-history":[{"count":1,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/posts\/1773\/revisions"}],"predecessor-version":[{"id":1783,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/posts\/1773\/revisions\/1783"}],"wp:attachment":[{"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/media?parent=1773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/categories?post=1773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perso.uclouvain.be\/tom.barbette\/wp-json\/wp\/v2\/tags?post=1773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}