[{"data":1,"prerenderedAt":522},["ShallowReactive",2],{"page-\u002Fblog\u002Fcompression-deep-dive-data-2026":3},{"id":4,"title":5,"body":6,"date":507,"description":508,"extension":509,"featured":510,"image":511,"meta":512,"navigation":510,"path":513,"published":510,"seo":514,"stem":515,"tags":516,"__hash__":521},"blog\u002Fblog\u002Fcompression-deep-dive-data-2026.md","Compression Benchmark Deep Dive: Every Algorithm, Every Dataset, Every Number",{"type":7,"value":8,"toc":490},"minimark",[9,19,35,61,66,69,130,134,137,141,144,149,156,159,193,197,200,203,214,221,225,228,235,246,249,253,256,270,274,277,280,313,317,320,337,341,344,449,459,463,466,481,484],[10,11,12,13,18],"p",{},"The ",[14,15,17],"a",{"href":16},"\u002Fblog\u002Fcompression-benchmark-real-files-2026","first post"," gave the narrative read. This one is the spreadsheet view: every algorithm next to its sweet spot, the heatmap that maps codecs to data categories, the full table of all 36 datasets, and a preset leaderboard you can sort however you want.",[10,20,21,22,26,27,30,31,34],{},"If you came here for the practical picks, the ",[23,24,25],"strong",{},"Algorithm sweet spots"," card grid below is the answer. If you came to argue with the data, scroll to the ",[23,28,29],{},"per-dataset table"," and the ",[23,32,33],{},"leaderboard",".",[36,37,39],"info",{"title":38},"What's in this post",[10,40,41,44,45,48,49,52,53,56,57,60],{},[23,42,43],{},"3,420"," measured runs · ",[23,46,47],{},"36"," real datasets · ",[23,50,51],{},"6"," algorithms · ",[23,54,55],{},"95"," distinct codec presets · ",[23,58,59],{},"27.3 GiB"," of source material covering codebases, single source files, documents, databases, binaries, images, audio, and video.",[62,63,65],"h2",{"id":64},"at-a-glance","At a glance",[10,67,68],{},"The headline answer to \"which codec should I use\" is anti-climactic: it depends on the data and what you're optimizing for. The more interesting answer comes out of the data:",[70,71,72,83,93,107,119],"ul",{},[73,74,75,78,79,82],"li",{},[23,76,77],{},"Brotli"," wins ",[23,80,81],{},"18 \u002F 36"," best-ratio matchups — far more than its reputation suggests, especially on small structured text and dense web-style assets.",[73,84,85,88,89,92],{},[23,86,87],{},"LZMA2"," takes ",[23,90,91],{},"10 \u002F 36"," best-ratio wins, mostly on heavy text-like archives where you can afford the time penalty.",[73,94,95,98,99,102,103,106],{},[23,96,97],{},"zstd"," and ",[23,100,101],{},"brotli"," tie at ",[23,104,105],{},"11 \u002F 36"," for best balanced score, the metric that actually matters for systems that decompress as much as they compress.",[73,108,109,110,114,115,118],{},"The single highest ratio in the entire benchmark is ",[111,112,113],"code",{},"brotli 11"," on the PDF sample at ",[23,116,117],{},"42.86x"," — which tells you more about that PDF than about Brotli.",[73,120,121,122,125,126,129],{},"The single best balanced score is ",[111,123,124],{},"zstd fast-3"," on a DOCX file at ",[23,127,128],{},"0.994"," — basically perfect on that workload because the DOCX was already a ZIP, so the fastest preset that doesn't hurt the ratio wins.",[62,131,133],{"id":132},"the-deep-dive","The Deep Dive",[135,136],"compression-deep-dive",{},[62,138,140],{"id":139},"how-to-read-the-data","How to read the data",[10,142,143],{},"The five visualizations above answer five different questions. They're meant to be read together, but each one stands alone.",[145,146,148],"h3",{"id":147},"the-sweet-spot-cards","The sweet-spot cards",[10,150,151,152,155],{},"For each codec, the card shows four presets: the strongest one, the most balanced one, the fastest one, and the smallest level that still captures ",[23,153,154],{},"95%"," of the codec's peak weighted ratio. That last one is the only honest answer to \"what should I actually use\" because it tells you when going higher stops being worth it.",[10,157,158],{},"A few patterns stand out:",[70,160,161,171,177,183],{},[73,162,163,166,167,170],{},[23,164,165],{},"zstd's"," strongest preset is ",[111,168,169],{},"22",", but its 95% sweet spot is much earlier in the level ladder. Most of zstd's value lives in the low-to-mid levels.",[73,172,173,176],{},[23,174,175],{},"Brotli's"," 95% sweet spot is high on the ladder. Brotli compresses more with extra time, but its top levels become very expensive on large directories.",[73,178,179,182],{},[23,180,181],{},"LZ4's"," strongest preset is barely stronger than its fastest one. That's the entire LZ4 thesis: speed first, ratio second.",[73,184,185,188,189,192],{},[23,186,187],{},"bzip2"," has only ",[111,190,191],{},"9"," sensible levels and its strongest one isn't dramatically stronger than its mid-levels — there's not much to tune.",[145,194,196],{"id":195},"the-category-heatmap","The category heatmap",[10,198,199],{},"The heatmap is the picture I refer to most. Each cell is \"this codec's best result on this category.\" Bright cells mean a codec absolutely loved that data shape. Dark cells mean it wasted CPU.",[10,201,202],{},"The shape of the heatmap is brutal:",[70,204,205,208,211],{},[73,206,207],{},"The top-left quadrant (codebases, source files, documents) is bright across most codecs.",[73,209,210],{},"The bottom rows (audio and video) are nearly flat regardless of codec.",[73,212,213],{},"The \"binary\" row is wildly mixed because the category itself is wildly mixed: random noise, VM disks, model weights, and package caches all live there.",[10,215,216,217,220],{},"Switch the metric to ",[23,218,219],{},"Compression throughput"," and a different story shows up: LZ4 dominates almost everything, which makes the speed-vs-ratio trade-off concrete instead of theoretical.",[145,222,224],{"id":223},"the-efficiency-frontier","The efficiency frontier",[10,226,227],{},"Each curve traces a single algorithm across its full level ladder. The x-axis is total compression time on a log scale.",[10,229,230,231,234],{},"The bend in every curve is the same point I mentioned in the first post: the place where extra time stops buying meaningful ratio. The interesting observation is ",[23,232,233],{},"how early that bend is",":",[70,236,237,240,243],{},[73,238,239],{},"For zstd, the curve is mostly flat after about level 3-7.",[73,241,242],{},"For Brotli, the bend is later — usually around level 7-9 — but the time cost climbs steeply after that.",[73,244,245],{},"For LZMA2, the curve climbs almost the whole way, which is why it wins so many ratio contests but takes so long.",[10,247,248],{},"Switch the y-axis to \"Space saved %\" and you can see the diminishing returns directly: going from 60% saved to 70% saved is much easier than going from 70% to 75%.",[145,250,252],{"id":251},"throughput-and-wins-by-family","Throughput and wins-by-family",[10,254,255],{},"Two complementary views:",[70,257,258,264],{},[73,259,260,263],{},[23,261,262],{},"Fastest preset throughput"," shows the absolute speed ceiling per family. LZ4 sits at the top; bzip2 sits at the bottom. Decompression is consistently faster than compression for every codec.",[73,265,266,269],{},[23,267,268],{},"Wins by family"," stacks \"best ratio\" wins and \"best balanced\" wins per algorithm. The shape is the headline: Brotli dominates raw ratio, zstd and Brotli tie on balanced score, and LZ4 quietly picks up balanced wins on data that nobody can compress well anyway.",[145,271,273],{"id":272},"the-full-per-dataset-table","The full per-dataset table",[10,275,276],{},"This is the data view I would copy out into a spreadsheet if I were planning real storage policy. Every row is one dataset; every cell is the actual measurement. Sort by size if you want to focus on the biggest workloads, sort by ratio to see which inputs are easy targets, sort by saved % to spot the surprises.",[10,278,279],{},"A few specific datasets are worth highlighting:",[70,281,282,288,294,304],{},[73,283,284,287],{},[23,285,286],{},"The PDF document"," is the highest-ratio result in the benchmark by a wide margin. That number tells you nothing about other PDFs — it tells you that this PDF sample contained large, repetitive, compressible streams.",[73,289,290,293],{},[23,291,292],{},"The journal log"," saves over 76% at the top result. Logs and codebases are the easiest, most predictable wins in the entire dataset.",[73,295,296,299,300,303],{},[23,297,298],{},"The Qwen3.5 LLM model directory"," is a 9.3 GiB binary tree where bzip2 took the best ratio. The same dataset has ",[111,301,302],{},"lz4 1"," as the best balanced pick because the absolute time cost of getting that ratio is enormous.",[73,305,306,30,309,312],{},[23,307,308],{},"The random binary blob",[23,310,311],{},"HEVC\u002FMP4\u002FMOV samples"," have ratios stuck at ~1.000x or even below. That's the benchmark saying \"do not even try.\"",[145,314,316],{"id":315},"the-preset-leaderboard","The preset leaderboard",[10,318,319],{},"Ninety-five presets, sortable. The defaults order by weighted ratio so you see who took the absolute crown. Click \"Throughput\" to find the speed kings. Click \"Balanced\" to find the actual everyday picks.",[10,321,322,323,326,327,329,330,329,333,336],{},"The most interesting filter is ",[23,324,325],{},"\"Wins\""," — how many of the 36 datasets a single preset took outright as best ratio. Almost every preset is at zero or one. A handful — ",[111,328,113],{},", ",[111,331,332],{},"lzma2 9e",[111,334,335],{},"lzma2 8e"," — concentrate the wins. That's a useful piece of intuition: the absolute leaderboard is dominated by a few brutal-but-slow presets, and almost everything else is a trade-off.",[62,338,340],{"id":339},"practical-takeaways-from-the-data","Practical takeaways from the data",[10,342,343],{},"If I had to compress these tables and charts down to one practical answer per workload, this is it:",[345,346,347,363],"table",{},[348,349,350],"thead",{},[351,352,353,357,360],"tr",{},[354,355,356],"th",{},"Workload",[354,358,359],{},"Use this",[354,361,362],{},"Why",[364,365,366,380,397,409,421,432],"tbody",{},[351,367,368,372,377],{},[369,370,371],"td",{},"Day-to-day developer compression",[369,373,374],{},[111,375,376],{},"zstd 3",[369,378,379],{},"Right at the bend in zstd's efficiency curve. Strong ratio, very fast, ubiquitous.",[351,381,382,385,394],{},[369,383,384],{},"Long-term backup of source code, logs, configs",[369,386,387,390,391],{},[111,388,389],{},"zstd 19"," or ",[111,392,393],{},"lzma2 7",[369,395,396],{},"Top-quartile ratio without extreme time cost. LZMA2 if you genuinely never decompress.",[351,398,399,402,406],{},[369,400,401],{},"Web text assets served behind a CDN",[369,403,404],{},[111,405,113],{},[369,407,408],{},"Pre-compressed once, served forever. Decompression is not the bottleneck.",[351,410,411,414,418],{},[369,412,413],{},"Real-time pipelines, fast local packaging",[369,415,416],{},[111,417,302],{},[369,419,420],{},"Decompression speed is the entire point.",[351,422,423,426,429],{},[369,424,425],{},"Already compressed media (mp4, hevc, mp3, aac, png)",[369,427,428],{},"Skip it",[369,430,431],{},"The data shows you'll lose CPU and gain bytes.",[351,433,434,437,446],{},[369,435,436],{},"Mixed-shape archives",[369,438,439,442,443],{},[111,440,441],{},"zstd 1","–",[111,444,445],{},"zstd 6",[369,447,448],{},"Any of these will be within striking distance of optimal on every category that matters.",[36,450,452],{"title":451},"A reminder about averages",[10,453,454,455,458],{},"Every aggregate number in this post is computed across ",[23,456,457],{},"all 36 datasets",", weighted by source size. That means a single very large dataset (the LLM model, the VM image, the random binary blob) can dominate weighted averages. The per-dataset table is the right place to go when you want the per-input answer rather than the global one.",[62,460,462],{"id":461},"whats-next","What's next",[10,464,465],{},"I want to extend the corpus in two directions:",[467,468,469,475],"ol",{},[73,470,471,474],{},[23,472,473],{},"More categories"," — game saves, sqlite databases under load, container image layers, parquet files. The \"binary\" bucket is currently doing too much work.",[73,476,477,480],{},[23,478,479],{},"Memory and CPU footprint"," — right now the benchmark only measures time and size. Peak resident memory and core-utilization would change the picture for some codecs significantly, especially LZMA2.",[10,482,483],{},"If there's a workload you'd want me to add, point me at a representative file and I'll fold it into the next round.",[10,485,486,487,34],{},"The companion narrative post is here if you want the prose read: ",[14,488,489],{"href":16},"Compression on Real Files: 3,420 Benchmark Runs",{"title":491,"searchDepth":492,"depth":492,"links":493},"",2,[494,495,496,505,506],{"id":64,"depth":492,"text":65},{"id":132,"depth":492,"text":133},{"id":139,"depth":492,"text":140,"children":497},[498,500,501,502,503,504],{"id":147,"depth":499,"text":148},3,{"id":195,"depth":499,"text":196},{"id":223,"depth":499,"text":224},{"id":251,"depth":499,"text":252},{"id":272,"depth":499,"text":273},{"id":315,"depth":499,"text":316},{"id":339,"depth":492,"text":340},{"id":461,"depth":492,"text":462},"2026-05-07T12:00:00+02:00","A data-first companion to the compression benchmark — sweet-spot picks per algorithm, an algorithm × category heatmap, the full per-dataset table, and a sortable preset leaderboard built from 3,420 measured runs.","md",true,null,{},"\u002Fblog\u002Fcompression-deep-dive-data-2026",{"title":5,"description":508},"blog\u002Fcompression-deep-dive-data-2026",[517,518,519,520,97,101],"compression","benchmark","data","performance","djLfdW2dmPCK_BUvsyTLaLuHoabJSCrkENqxqwpIZv4",1780404293374]