- Take a large lossless image like this one and cut it up in a not too small number of large tiles. In my test, I have cropped the image a bit to get a 5,000 x 5,000 pixel image, then split it in 25 tiles of 1,000 x 1,000 pixels.
- Compress all the tiles to the JPEG format and measure the lossiness of each one (I have used the PSNR value)
- For each tile, compress the original using WebP with different values for the quality parameter, until for each tile you get a WebP image whose PSNR value is just as good as that of the JPEG image obtained in the previous step. What we are doing is getting images with very similar level of lossiness. PSNR values will never be the same. I have given the JPEG tiles a small advantage by requiring a better PSNR value from the WebP tiles.
- Compute the size ratio for each pair of tiles as [size of the WebP file] / [size of the JPEG file]
- Take the resulting ratios and compute their average and standard deviation. We can say that WebP is significantly better than JPEG iff the average plus twice the standard deviation is lower than 1, because we’ll be able to say that in 95% of the cases, the WebP image will be smaller than the equivalent (in terms of lossiness) JPEG image. If the average plus twice the standard deviation is bigger than 1, then we’ll have to say that these formats have similar compression performance. This would be a case where compression performances are not significantly different:
These are the results of my test (click to enlarge):
As we can see, the average plus twice the standard deviation is below 1:
- According to this test, the WebP format is about 6% better than JPEG. We can say that in 95% of the cases, the resulting image will be at least 6% smaller than the equivalent JPEG image.
- Of course this test has important flaws: we have only tested one type of image (aerial orthophoto), the number of tiles is not very high (25), the lossiness of the compared images was not exactly the same, etc, but I think the result is a good indication of how these two formats compare.
- We can see that the size ratios do follow the Normal distribution pattern, so using the average plus twice the standard deviation value was not a bad idea.
- My opinion is that this is a rather poor result for the WebP format, and it will not replace JPEG as the de facto standard for image delivery on the Web.