@c -*-texinfo-*-

@node compression
@section Data decompression

The libraries in this section deal with data decompression. They're
currently all based around the INFLATE algorithm that decompresses
data created by gzip, zip and zlib.

@menu
* compression adler-32::       Mark Adler's Adler-32 checksum
* compression gzip::           GZIP custom input port
@c * compression huffman::        Huffman code utilities
* compression inflate::        Decompress DEFLATE'd data
* compression sliding-buffer:: A circular buffer attached to a data sink
* compression xz::             XZ custom input port
* compression zip::            ZIP archive reader/writer
* compression zlib::           ZLIB custom input port
@end menu

@c ======================================================================
@node compression adler-32
@subsection Mark Adler's Adler-32 checksum
The @code{(weinholt compression adler-32)} library provides the
Adler-32 checksum used in the ZLIB data format. @xref{compression zlib}. 

The procedures defined are similar to those made by @code{define-crc}
in @ref{crypto crc}. The exported bindings are @code{adler-32},
@code{adler-32-init}, etc.
@cindex Adler-32

@noindent
Version history:
@itemize
@item
@code{(0 0)} -- Initial version
@end itemize

@c ======================================================================
@node compression gzip
@subsection GZIP custom input port
The @code{(weinholt compression gzip)} library provides a custom
input port for reading GZIP compressed data.

A word of warning: the current implementation uses an internal buffer
that can grow big when reading specially crafted data.

The GZIP format can support multiple compression methods, but only
DEFLATE'd data is supported in this library.

@deffn Procedure is-gzip-file? filename-or-port
Takes a filename or a binary input port and returns true if the file
looks like a GZIP file. The port should have @code{set-port-position!}
and @code{port-position}.
@end deffn

@deffn Procedure make-gzip-input-port binary-input-port id close-underlying-port?
Returns a new port that can be used to read decompressed GZIP data
from the @var{binary-input-port}. The @var{id} is the name of the
returned port. 

If @var{close-underlying-port?} is true then at the end of the GZIP
stream the @var{binary-input-port} will be closed.
@end deffn

@deffn Procedure open-gzip-file-input-port filename
Opens the file specified by @var{filename} and returns a binary input
port that decompresses the file on-the-fly.
@end deffn

@deffn Procedure extract-gzip binary-input-port binary-output-port
Reads compressed data from @var{binary-input-port} and writes the
decompressed data to @var{binary-output-port}. Returns a list of gzip
headers, one for each gzip member of the file (gzip files can be
concatenated).
@end deffn

@deffn Procedure get-gzip-header binary-input-port
Reads a GZIP header from @var{binary-input-port} and performs sanity
checks.
@end deffn

See RFC 1952 for a full description of the following values. Updates
are available at @uref{http://www.gzip.org/format.txt}.

@deffn Procedure gzip-text? hdr
True if the uncompressed data associated with @var{hdr} is believed to
be text.
@end deffn

@deffn Procedure gzip-mtime hdr
The file's modification time as an SRFI-19 date or @code{#f} is none
is available.
@end deffn

@deffn Procedure gzip-extra-data hdr
An ``extra field'' which some systems use to encode additional file
attributes. This is an unparsed bytevector.
@end deffn

@deffn Procedure gzip-filename hdr
The file's original filename as a string or @code{#f} if none is
available.
@end deffn

@deffn Procedure gzip-comment hdr
A file comment as a string or @code{#f} if none is available.
@end deffn

@deffn Procedure gzip-method hdr
The symbol @code{slowest}, @code{fastest} or an integer (probably
denoting a different compression setting).
@end deffn

@deffn Procedure gzip-os hdr
The id number of the operating system that created the file. It is
e.g.@: 0 for DOS, 1 for Amiga, 2 for VMS, 3 for Unix.
@end deffn

@noindent
Version history:
@itemize
@item
@code{(0 0)} -- Initial version.
@item
@code{(1 0)} -- GZIP headers are returned as a record type.
@code{extract-gzip} returns a list of headers.
@end itemize

@c ======================================================================
@node compression inflate
@subsection Decompress DEFLATE'd data
The procedures in @code{(weinholt compression inflate)} decompress
DEFLATE data streams. DEFLATE is the data format used by gzip, zip and
zlib. This library handles the raw data stream.

@deffn Procedure inflate binary-input-port binary-output-port @
                         crc-init crc-update crc-finish
Inflates a complete DEFLATE data stream. It reads compressed data from
@var{binary-input-port} and writes decompressed data to
@var{binary-output-port}.

The arguments @var{crc-init}, @var{crc-update} and @var{crc-finish}
should have the same semantics that @code{(weinholt crypto crc)} uses,
see @ref{crypto crc} and @ref{compression adler-32}.

Three values are returned: the final CRC of the decompressed data, its
length, and a bytevector with read but unused bytes from the input.
@end deffn

@deffn Procedure make-inflater binary-input-port sink window-size dictionary
Returns a procedure that, when called, decompresses a DEFLATE block
from @var{binary-input-port}. The returned procedure should be called
with zero arguments and returns either the symbol @code{done}, to
signify the end of the DEFLATE stream, or @code{more} to indicate more
blocks are (or will be) available.

For a description of the @var{sink} argument,
@pxref{compression sliding-buffer}.

The @var{window-size} is the size of the sliding window buffer. The
most common value is @math{32*1024} bytes, but each DEFLATE stream has
a correct value that was used when creating the stream. For zlib
streams this value is specified in the header.

The @var{dictionary} is a bytevector that is prepended to the output
buffer, but it is not actually copied to the output.
@xref{compression zlib}.

The inflate algorithm needs some lookahead and therefore it can read a
byte or two that does not belong to the inflate stream itself. Apply
the symbol @code{get-buffer} to the returned procedure to recover
those extra bytes as a bytevector.
@end deffn

@noindent
Version history:
@itemize
@item
@code{(0 0)} -- Initial version
@item
@code{(1 0)} -- @code{inflate} returns three values (backwards
incompatible change). The inflater procedures gain a way to return any
buffered bytes.
@end itemize

@c ======================================================================
@node compression sliding-buffer
@subsection A circular buffer attached to a data sink
The @code{(weinholt compression sliding-buffer)} library provides a
circular buffer that passes the buffered data to a sink (a
@dfn{sliding window}).

A @dfn{sink} is a procedure with three arguments: a bytevector
@var{bv}, an integer @var{start} and an integer @var{count}. When the
sink procedure is called it should process @var{count} bytes starting
at offset @var{start} of @var{bv}.

This library was written by Andreas Rottmann (and has been modified,
see the source code for a history). It is used by the @code{(weinholt
compression inflate)} library because the LZ77 component in INFLATE
needs a way to copy data that has already been written to the output,
and this data structure obviates the need to use a file for that
purpose.

@deffn Procedure make-sliding-buffer sink size
Returns a new sliding buffer with the given @var{sink} and @var{size}.
The @var{size} determines how far back in the output stream
@code{sliding-buffer-dup!} can look.
@end deffn

@deffn Procedure sliding-buffer? obj
True if @var{obj} is a sliding buffer.
@end deffn

@deffn Procedure sliding-buffer-init! buffer bv
Copy initial data into the buffer so that it can be used with
@code{sliding-buffer-dup!}. The sink does not receive this data.
@end deffn

@deffn Procedure sliding-buffer-drain! buffer
Sends the buffered data to to the @var{buffer}'s sink.
@end deffn

@deffn Procedure sliding-buffer-read! buffer binary-input-port len
Reads @var{len} bytes from @var{binary-input-port} into the
@var{buffer}.
@end deffn

@deffn Procedure sliding-buffer-put-u8! buffer u8
Copies the byte @var{u8} into the @var{buffer}.
@end deffn

@deffn Procedure sliding-buffer-dup! buffer distance len
Duplicates @var{len} bytes from inside the output stream of
@var{buffer} at @var{distance} bytes from the current end of the
buffer.
@end deffn

@c ======================================================================
@node compression xz
@subsection XZ custom input port
The @code{(weinholt compression xz)} library provides a custom input
port for reading XZ compressed data. XZ is a wrapper format around the
LZMA2 algorithm and it is becoming popular as a gzip/bzip2
replacement.

@emph{Note}: An XZ file can specify several types of filters, other
than LZMA2, but these have currently not been implemented.

The LZMA2 algorithm uses a sliding buffer that may be up to 4
gigabytes. This might cause problems when reading XZ files.

@deffn Procedure is-xz-file? filename-or-port
Takes a filename or a binary input port and returns true if the file
looks like a XZ file. The port should have @code{set-port-position!}
and @code{port-position}.
@end deffn

@deffn Procedure make-xz-input-port binary-input-port id close-underlying-port?
Returns a new port that can be used to read decompressed XZ data
from the @var{binary-input-port}. The @var{id} is the name of the
returned port. 

To verify that the file was decompressed correctly it is necessary to
close the port. On close the port will read all remaining data and
compare its checksum to the checksum at the end of the file.

If @var{close-underlying-port?} is true then when the XZ input port is
closed the @var{binary-input-port} will also be closed.
@end deffn

@deffn Procedure open-xz-file-input-port filename
Opens the file specified by @var{filename} and returns a binary input
port that decompresses the file on-the-fly.
@end deffn


@noindent
Version history:
@itemize
@item
@code{(1 0)} -- Initial version.
@end itemize

@c ======================================================================
@node compression zip
@subsection ZIP archive reader/writer
The @code{(weinholt compression zip)} library provides procedures for
reading and writing ZIP archives.

This library exports bindings that aren't easily identified as having
to do with ZIP archives, so I suggest you use a prefix as described in
@ref{Conflicting names}.

The @code{(weinholt compression zip extra)} library is used to set and
retrieve file attributes, look for absolute/relative path attacks,
create directories, and handle system-specific file types. None of
this can really be done portably, so the default version of that
library does the minimum possible. A few implementation-dependent
overrides are included which allow directories to be created and
handle some attributes.

To learn about the file format, see
@uref{http://www.info-zip.org/doc/}. In brief: each file has a file
record (followed by the file data), and the archive ends with a list
of central directory records and a special end of central directory
record. Some information is duplicated in the file and central
directory records.

@deffn Procedure get-central-directory binary-input-port
Returns the central directory of the ZIP archive in
@var{binary-input-port}. This is a list of central directory records
and the end of central directory record.
@end deffn

@deffn Procedure central-directory->file-record zip-port cdir
Uses the data in the central directory record @var{cdir} to read the
associated file record from @var{zip-input-port}. The returned value
is also referred to as a @dfn{local file header}.
@end deffn

@deffn Procedure extract-file zip-port local central
Extracts the file associated with the @var{local} and @var{central}
records. The @var{zip-port} is the same port the records were read
from.

The extracted file will be created relative to the current
working directory (or default filespec) and will retain as many
attributes as possible from those recorded in the ZIP archive.
@end deffn

@deffn Procedure extract-to-port zip-port local central dest-port
Extracts the file associated with the @var{local} and @var{central}
records to the given binary output port @var{dest-port}. The
@var{zip-port} is the same port the records were read from.

It is possible to preserve the file's attributes (at least if the
extracted file is a regular byte stream) by using the accessors for
@var{local} and @var{central} similarly to how the ``extra'' library
uses that data.
@end deffn

Creating a ZIP archive is done by appending each file, and then when
done appending the central directory. The central directory is in this
case a list of central directory records returned by e.g.@:
@code{append-file}. The port the ZIP archive is written to must
support @code{port-position} and @code{set-port-position!}.

@emph{Note}: Currently there is no compression performed when creating
archives.

@deffn Procedure append-file zip-port filename
Appends the file given by @var{filename} to @var{zip-port}, which is a
binary output port. Returns a central directory record.
@end deffn

@deffn Procedure append-port zip-port data-port filename date @
                       local-extra central-extra os-made-by @
                       internal-attributes external-attributes
Similar to @code{append-file}, except no file is used. Instead the
data for the file is read from the binary input port @var{data-port}.
Because there is no file, all the file attributes need to be provided
explicitly. A central directory record is returned.

For a description of the attributes, see the accessors for file and
central directory records.
@end deffn

@deffn Procedure append-central-directory zip-port centrals
Writes a list of central directory records to the @var{zip-port} and
then appends the special @emph{end of central directory record}. After
this no more data should be written to the ZIP archive. The list of
central directory records @code{centrals} should be those returned by
@code{append-file} and @code{append-port}.
@end deffn

@deffn Procedure create-file zip-port filenames
Builds a complete ZIP archive that includes all the files specified by
the list @var{filenames} and writes it to @var{port}, which should be
a binary output port.
@end deffn

@deffn Procedure supported-compression-method? n
True if @var{n} represents a supported compression method. Currently
only stored and deflated are supported. See
@code{file-record-compression-method}.
@end deffn

@deffn Procedure unsupported-error? obj
If an attempt was made to access an unsupported file record or to
extract a file using an unsupported compression method then a
condition will be raised that satisfies this predicate.
@end deffn

@deffn Procedure file-record? obj
True if @var{obj} is a file record.
@end deffn

@deffn Procedure file-record-minimum-version frec
This is the minimum supported version of the ZIP standard required to
extract the file. Currently vresion 2.0 is supported (which is encoded
as the exact integer 20).
@end deffn

@deffn Procedure file-record-flags frec
Various flags that can indiciate which compression option was used,
etc. You can probably ignore these.
@end deffn

@deffn Procedure file-record-compression-method frec
Returns an integer that represents the compression method that was
used when storing the file associated with @var{frec}. Most ZIP files
use only Deflate and store.

@itemize
@item
@code{compression-stored} means the file was stored without any
compression.
@item
@code{compression-shrunk} is the obsolete Shrunk method.
@item
@code{compression-reduced1} is the obsolete Reduced method with
factor 1.
@item
@code{compression-reduced2} same as above, factor 2.
@item
@code{compression-reduced3} same as above, factor 3.
@item
@code{compression-reduced4} same as above, factor 4.
@item
@code{compression-imploded} is the obsolete Implode method.
@item
@code{compression-deflated} is the Deflate compression algorithm.
@item
@code{compression-deflate64} is a slightly modified Deflate.
@item
@code{compression-pkimplode} is something else.
@item
@code{compression-bzip2} is BZIP2.
@end itemize
@end deffn

@deffn Procedure file-record-date frec
The file's modification time as an SRFI-19 date.
@end deffn

@deffn Procedure file-record-crc-32 frec
The file's CRC-32 checksum. @xref{crypto crc}.
@end deffn

@deffn Procedure file-record-compressed-size frec
The number of bytes the file uses inside the ZIP archive.
@end deffn

@deffn Procedure file-record-uncompressed-size frec
The number of bytes the file will use when it has been decompressed.
@end deffn

@deffn Procedure file-record-filename frec
The filename of the file. This might be different in the associated
central directory record (e.g.@: due to mischief). This can also be
the string @code{"-"} if the file came from the standard input port.
@end deffn

@deffn Procedure file-record-extra frec
An list of id and data pairs. This is used to encode file attributes,
etc. See the file format specification for more information.
@end deffn

@c @deffn Procedure file-record-data-port-position frec
@c The position of the file's data inside the ZIP archive.
@c @end deffn

@deffn Procedure central-directory? obj
True if @var{obj} is a central-directory record.
@end deffn

@deffn Procedure central-directory-version-made-by cdir
This is the version of the ZIP standard supported by the
implementation that created the archive.
@end deffn

@deffn Procedure central-directory-os-made-by cdir
The ID number of the operating system on which the ZIP archive was
created. See the file format specification for a full list (DOS is 0,
Unix is 3).
@end deffn

@deffn Procedure central-directory-minimum-version cdir
This is the minimum supported version of the ZIP standard required to
extract the file. Currently version 2.0 is supported (which is encoded
as the exact integer 20).
@end deffn

@deffn Procedure central-directory-flags cdir
See @code{file-record-flags}.
@end deffn

@deffn Procedure central-directory-compression-method cdir
See @code{file-record-compression-method}.
@end deffn

@deffn Procedure central-directory-date cdir
The file's modification time as an SRFI-19 date.
@end deffn

@deffn Procedure central-directory-crc-32 cdir
The file's CRC-32 checksum. @xref{crypto crc}.
@end deffn

@deffn Procedure central-directory-compressed-size cdir
The number of bytes the file uses inside the ZIP archive.
@end deffn

@deffn Procedure central-directory-uncompressed-size cdir
The number of bytes the file will use when it has been decompressed.
@end deffn

@deffn Procedure central-directory-disk-number-start cdir
The number of the split archive that the file starts on. Note that
there is no explicit support for split archives, so this is untested.
@end deffn

@deffn Procedure central-directory-internal-attributes cdir
Bit 0 of this integer is set if the file is believed to be text. This
might be useful for end of line conversion, but it is probably
unreliable.
@end deffn

@deffn Procedure central-directory-external-attributes cdir
The file attributes of the file. The format depends on the os-made-by field.
@end deffn

@c @deffn Procedure central-directory-local-header-offset cdir
@c @end deffn

@deffn Procedure central-directory-filename cdir
See @code{file-record-filename}.
@end deffn

@deffn Procedure central-directory-extra cdir
See @code{file-record-extra}. Note that some of the fields have the
same ID here and in the file records, but slightly different
encodings.
@end deffn

@deffn Procedure central-directory-comment cdir
A textual comment associated with the file.
@end deffn

@deffn Procedure end-of-central-directory? obj
True of @var{obj} is an end-of-central-directory record.
@end deffn

@deffn Procedure end-of-central-directory-disk edir
The number of the split archive where @var{edir} is located.
@end deffn

@deffn Procedure end-of-central-directory-start-disk edir
The number of the split archive where the central directory begins.
@end deffn

@deffn Procedure end-of-central-directory-entries edir
The number of records in the central directory in this split archive.
@end deffn

@deffn Procedure end-of-central-directory-total-entries edir
The number of records in the central directory for the whole archive.
@end deffn

@c @deffn Procedure end-of-central-directory-size edir
@c @end deffn

@c @deffn Procedure end-of-central-directory-offset edir
@c @end deffn

@deffn Procedure end-of-central-directory-comment edir
A textual comment associated with the whole archive.
@end deffn

@noindent
Version history:
@itemize
@item
@code{(0 0)} -- Initial version
@end itemize

@c ======================================================================
@node compression zlib
@subsection ZLIB custom input port
The @code{(weinholt compression zlib)} library provides a custom
input port for reading ZLIB compressed data.

@deffn Procedure make-zlib-input-port binary-input-port id max-buffer-size close-underlying-port? dictionaries
Returns a binary input port that decompresses and reads a ZLIB stream
from the binary input port @var{binary-input-port}. The @var{id} is
the name of the returned custom binary input port.

If @var{max-buffer-size} is false then the internal buffer can grow
without bounds (might be a bad idea). Protocols using ZLIB will
normally specify a "flush" behavior. If your protocol uses flushing
and specifies a maximum record size, then use that size as
max-buffer-size.

If @var{close-underlying-port?} is true then at the end of the zlib
stream the @var{binary-input-port} will be closed.

An application can define dictionaries which can improve compression
by containing byte sequences commonly found at the start of files. The
@var{dictionaries} argument is an alist that maps Adler-32 checksums
to bytevectors. @xref{compression adler-32}.
@end deffn

@noindent
Version history:
@itemize
@item
@code{(0 0)} -- Initial version
@end itemize
