[conspire] [png-mng-implement] Why is it that png files are not dependable over the long haul

[Cosmin Truta]

Hello, Ruben,

I'm sorry for your bad experience. Here is what happened:

Ruben Safir wrote:
> http://www.mrbrklyn.com/brooklyn/images/wall_art_river_and_metropolitian_avenue_williamsburg_2006_00578.png
> http://www.mrbrklyn.com/brooklyn/images/ornimental_building_kent_st_2006_00557.png
> [...]
> IDAT: invalid distance too far back

Both of these PNG files are broken, most likely produced by a faulty
encoder. In a few days, I hope to provide you with a special build of
OptiPNG to fix them, which I may later continue to implement into a
proper OptiPNG recovery feature.

It is also important to mention that the libpng is correct. Some
applications (like Gimp) refuse to load the images, others (like
Chrome) shows them just fine. They're all correct, in spite of their
differences. As odd as it may seem.

Here is why:

It's not even a PNG problem, but a problem in the zlib-compressed
datastream (RFC-1950), which the PNG format is enclosing. See the
message: "invalid distance too far back" is an error produced by the
zlib implementation. (Again: not by libpng.) Admittedly, not
user-friendly, and yet, perhaps frustratingly, entirely correct.

The PNG specification (and zlib, and deflate, etc.) have strict rules
for error detection, but no rules for error recovery. The same is true
about JPEG, GIF, TIFF, and maybe any other image codec out there, for
all I know. Error recovery is up to the PNG-supporting applications,
it's entirely their freedom. A web browser may choose to set up the
zlib decoder in a way that allows faulty zlib datastreams to be
silently accepted, because the implementors of that web browser may
consider it a useful behaviour for their users. Conversely, an image
converter may choose to detect those errors, because the implementors
of that converter may consider a useful behaviour to let the users
know if something is wrong in the image file. The libpng
implementation is set up, by default, to be strict, and, as a
consequence, implementations that do not override that setting will
fail with your images.

And then, there is a third category: applications like OptiPNG,
specifically designed to both signal the errors, and to recover from
them. It is impossible, in general, to know how to recover from every
conceivable kind of error. But for your specific case, the recovery is
obvious. Not easy to implement (which is why common applications don't
currently do it), but easy to understand, at least from a "historical"
point of view.

The explanation is as follows:

Long ago (more than 10 years ago, IIRC around the year 2004), there
was a bug in the libpng encoder that existed for a short period of
time, that produced zlib streams with an incorrect window size. By
incorrect, I mean smaller than they should be. They can be larger than
they should, that's ok; but smaller, that's not ok. Your images have a
small window size, so they are not ok. The pngcheck program indicates
an 8KB window, when in fact it should be 32KB.

The recovery consists in making the 32KB window, always, because that
is the maximum, then decode, and re-encode. That's easy to do. Also
easy to explain: use a larger array, problem solved.

But then you have a problem with spec compliance. You may now have
out-of-spec images (like yours), without knowing that they're
out-of-spec, because your larger zlib deflate window accepted them.
Who knows that's not data corruption, or some other evil attack? Well,
we know: we had a bug, long ago, which was fixed, also long ago, but a
few unlucky users ended up with corrupted images that need fixing.

Now: in order to fix, first you need to know the need to fix. In order
to know the need to fix, you need the original zlib window size, which
should cause the zlib error. You should recognize this zlib error, in
order to know what to recover. (The zlib API does not make this task
easy, which is unfortunate.) Then, in order to recover, you need to
rewind your zlib stream (or even the entire PNG stream), then grow the
zlib window size to the maximum 32KB, and then restart the decoder.
Then decode, then re-encode... Fixed!

This is doable, but tedious, and simple libpng-based applications are
neither required to do it, not customarily designed/implemented to do
it.

I encourage you to file an OptiPNG feature request for this sort of
error recovery.
https://sourceforge.net/p/optipng/feature-requests/

Sincerely,
Cosmin

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
png-mng-implement mailing list
png-mng-implement at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/png-mng-implement