Beware of BrotliStream in .NET Core 2.1

I love compression algorithms and have built a fair share of my own for fun and profit. It is a difficult task to build something that is efficient when it comes to the compromise between speed and compression ratio, so when I saw Google's Brotli algorithm mentioned in the .NET Core 2.1 RC1 announcement, I was estatic!

It lasted about 15 minutes until I put the BrotliStream to a test, only to find out that it took 140x longer to compress a 518 MB file!

Curious if this was a bug, I delved into the code and realized that the default compression level is set to 11. For those not familiar with Brotli, that is the highest level available, that prioritize compression ratio above speed.

I reported it as an issue, but Microsoft seems to think that having the same configuration as Google's Brotli library is more important than a sane default, so it is only a matter of time before tons of StackOverflow posts popup complaining about the speed of BrotliStream.

If you want to use Brotli in your application (a wise choice! it is a good algorithm), be sure to NOT use the builtin CompressionLevel enum in the BrotliStream constructor. Instead, cast an integer (I would recommend a value of 6) like this:

BrotliStream stream = new BrotliStream(sm, (CompressionLevel)6);

That is a good compromise between speed and compression ratio for most data.

Update 2021-11-01: Microsoft decided to block user settings in this merge request.

Update 2022-05-01: Microsoft now defaults to quality 4. It will be part of .NET 7.

Update 2022-09-01: Microsoft acknowledges the problem, but stays with the decision of setting quality to 4.

Comments

Post a Comment

Popular posts from this blog

.NET Compression Libraries Benchmark

Image
I often find myself needing a compression library in a software development project, and after a few tests with disappointing results I usually fall back to spawning 7Zip as an external process. This time however, the application specification prohibits any data written to disk, which 7Zip needs to do in order to work correctly. I needed to do everything in memory, and so I had to test some of the C# libraries available for compression as well as the different compression algorithms. Here is a full list of the tested libraries: .NET DeflateStream ICSharpCode SharpZip Zlib.NET DotNetZip Managed LZMA LZMA SDK QuickLZ    Compression 101 Before I get to the results of the benchmark, I think it is important to understand the different archive formats and compression algorithms in order to compare the results with other benchmarks/and or tests. Lets get to it! A compression algorithm can be compared to a small machine that takes in some data, do some math on it and transfor
Read more

Reducing the size of self-contained .NET Core applications

Just for note keeping, I've written down some methods of reducing the size of a .NET Core application. I thought others could use it as well, so here you go. Original Size First off, let's see how much disk space a self-contained 'hello world' application takes up. > dotnet new console > dotnet publish -r win-x86 -c release Size: 53.9 MB -  yuck! Trimming Microsoft has built a tool that finds unused assemblies and removes them from the distribution package. This is much needed since the 'netcoreapp2.0' profile is basically .NET Framework all over again, and it contains a truckload of assemblies our little 'hello world' application don't use. > dotnet new console > dotnet add package Microsoft.Packaging.Tools.Trimming -v 1.1.0-preview1-25818-01 > dotnet publish -r win-x86 -c release /p:TrimUnusedDependencies=true Size: 15.8 MB Much better! Although, we have only removed unused assemblies - what about unused
Read more

Convex polygon based collision detection

Image
A lot of people have asked me: Why use a convex polygon only engine like Box2D? In this post I will provide some info on why convex polygons is the optimal polygon type to use. What does convex mean? Convex polygons needs to satisfy the following requirements: All internal angles must be below 180 degrees Every line segment between two vertices can be contained fully inside or at the edge the polygon A non-convex polygon is called a concave polygon and it must have an internal angle measuring greater than 180 degrees. As you might have thought about yourself; concave polygons can be quite handy. Convex-only engines can’t represent a simple player polygon like Pac-Man . Why use convex-only? There are really only one big juicy reason: performance Physics engines are able to take quite a few shortcuts and use cheap algorithms to determine the closest vertex, shortest distance and a lot more. An algorithm like Separating Axis Theorem and the Gilbert – Johnson –
Read more