package zstd

Import Path
	github.com/klauspost/compress/zstd (on go.dev)

Dependency Relation
	imports 25 packages, and imported by 5 packages



Code Examples

	
		package main
		
		import (
			"bytes"
			"fmt"
		
			"github.com/klauspost/compress/zstd"
		)
		
		func main() {
			// "Raw" dictionaries can be used for compressed delta encoding.
		
			source := []byte(`
				This is the source file. Compression of the target file with
				the source file as the dictionary will produce a compressed
				delta encoding of the target file.`)
			target := []byte(`
				This is the target file. Decompression of the delta encoding with
				the source file as the dictionary will produce this file.`)
		
			// The dictionary id is arbitrary. We use zero for compatibility
			// with zstd --patch-from, but applications can use any id
			// not in the range [32768, 1<<31).
			const id = 0
		
			bestLevel := zstd.WithEncoderLevel(zstd.SpeedBestCompression)
		
			w, _ := zstd.NewWriter(nil, bestLevel,
				zstd.WithEncoderDictRaw(id, source))
			delta := w.EncodeAll(target, nil)
		
			r, _ := zstd.NewReader(nil, zstd.WithDecoderDictRaw(id, source))
			out, err := r.DecodeAll(delta, nil)
			if err != nil || !bytes.Equal(out, target) {
				panic("decoding error")
			}
		
			// Ordinary compression, for reference.
			w, _ = zstd.NewWriter(nil, bestLevel)
			compressed := w.EncodeAll(target, nil)
		
			// Check that the delta is at most half as big as the compressed file.
			fmt.Println(len(delta) < len(compressed)/2)
		}




Package-Level Type Names (total 8)


	/* sort by:  |  */
	

		
			
				CompatV155 will make the dictionary compatible with Zstd v1.5.5 and earlier.
				See https://github.com/facebook/zstd/issues/3724

			
				Content to use to create dictionary tables.

			
				DebugOut will write stats and other details here if set.

			
				History to use for all blocks.

			
				Dictionary ID.

			
				Use the specified encoder level.
				The dictionary will be built using the specified encoder level,
				which will reflect speed and make the dictionary tailored for that level.
				If not set SpeedBestCompression will be used.

			
				Offsets to use.

		
			func BuildDict(o BuildDictOptions) ([]byte, error)


	
		Decoder provides decoding of zstandard streams.
		The decoder has been designed to operate without allocations after a warmup.
		This means that you should store the decoder for best performance.
		To re-use a stream decoder, use the Reset(r io.Reader) error to switch to another stream.
		A decoder can safely be re-used even if the previous stream failed.
		To release the resources, you must call the Close() function on a decoder.

		
			
				Close will release all resources.
				It is NOT possible to reuse the decoder after this.

			
				DecodeAll allows stateless decoding of a blob of bytes.
				Output will be appended to dst, so if the destination size is known
				you can pre-allocate the destination slice to avoid allocations.
				DecodeAll can be used concurrently.
				The Decoder concurrency limits will be respected.

			
				IOReadCloser returns the decoder as an io.ReadCloser for convenience.
				Any changes to the decoder will be reflected, so the returned ReadCloser
				can be reused along with the decoder.
				io.WriterTo is also supported by the returned ReadCloser.

			
				Read bytes from the decompressed stream into p.
				Returns the number of bytes read and any error that occurred.
				When the stream is done, io.EOF will be returned.

			
				Reset will reset the decoder the supplied stream after the current has finished processing.
				Note that this functionality cannot be used after Close has been called.
				Reset can be called with a nil reader to release references to the previous reader.
				After being called with a nil reader, no other operations than Reset or DecodeAll or Close
				should be used.

			
				WriteTo writes data to w until there's no more data to write or when an error occurs.
				The return value n is the number of bytes written.
				Any error encountered during the write is also returned.

		
			*Decoder : github.com/gobwas/ws.HandshakeHeader
			*Decoder : io.Reader
			*Decoder : io.WriterTo
		
			func NewReader(r io.Reader, opts ...DOption) (*Decoder, error)


	
		DOption is an option for creating a decoder.

		
			func IgnoreChecksum(b bool) DOption
			func WithDecodeAllCapLimit(b bool) DOption
			func WithDecodeBuffersBelow(size int) DOption
			func WithDecoderConcurrency(n int) DOption
			func WithDecoderDictRaw(id uint32, content []byte) DOption
			func WithDecoderDicts(dicts ...[]byte) DOption
			func WithDecoderLowmem(b bool) DOption
			func WithDecoderMaxMemory(n uint64) DOption
			func WithDecoderMaxWindow(size uint64) DOption
		
			func NewReader(r io.Reader, opts ...DOption) (*Decoder, error)
			func ZipDecompressor(opts ...DOption) func(r io.Reader) io.ReadCloser
			func github.com/hamba/avro/v2/ocf.WithZStandardDecoderOptions(opts ...DOption) ocf.DecoderFunc


	
		Encoder provides encoding to Zstandard.
		An Encoder can be used for either compressing a stream via the
		io.WriteCloser interface supported by the Encoder or as multiple independent
		tasks via the EncodeAll function.
		Smaller encodes are encouraged to use the EncodeAll function.
		Use NewWriter to create a new instance.

		
			
				Close will flush the final output and close the stream.
				The function will block until everything has been written.
				The Encoder can still be re-used after calling this.

			
				EncodeAll will encode all input in src and append it to dst.
				This function can be called concurrently, but each call will only run on a single goroutine.
				If empty input is given, nothing is returned, unless WithZeroFrames is specified.
				Encoded blocks can be concatenated and the result will be the combined input stream.
				Data compressed with EncodeAll can be decoded with the Decoder,
				using either a stream or DecodeAll.

			
				Flush will send the currently written data to output
				and block until everything has been written.
				This should only be used on rare occasions where pushing the currently queued data is critical.

			
				MaxEncodedSize returns the expected maximum
				size of an encoded block or stream.

			
				ReadFrom reads data from r until EOF or error.
				The return value n is the number of bytes read.
				Any error except io.EOF encountered during the read is also returned.
				
				The Copy function uses ReaderFrom if available.

			
				Reset will re-initialize the writer and new writes will encode to the supplied writer
				as a new, independent stream.

			
				ResetContentSize will reset and set a content size for the next stream.
				If the bytes written does not match the size given an error will be returned
				when calling Close().
				This is removed when Reset is called.
				Sizes <= 0 results in no content size set.

			
				Write data to the encoder.
				Input data will be buffered and as the buffer fills up
				content will be compressed and written to the output.
				When done writing, use Close to flush the remaining output
				and write CRC if requested.

		
			*Encoder : github.com/apache/thrift/lib/go/thrift.Flusher
			*Encoder : github.com/miekg/dns.Writer
			*Encoder : github.com/parquet-go/parquet-go/compress.Writer
			*Encoder : github.com/prometheus/common/expfmt.Closer
			*Encoder : internal/bisect.Writer
			*Encoder : io.Closer
			*Encoder : io.ReaderFrom
			*Encoder : io.WriteCloser
			*Encoder : io.Writer
		
			func NewWriter(w io.Writer, opts ...EOption) (*Encoder, error)


	
		EncoderLevel predefines encoder compression levels.
		Only use the constants made available, since the actual mapping
		of these values are very likely to change and your compression could change
		unpredictably when upgrading the library.

		
			
				String provides a string representation of the compression level.

		
			 EncoderLevel : expvar.Var
			 EncoderLevel : fmt.Stringer
		
			func EncoderLevelFromString(s string) (bool, EncoderLevel)
			func EncoderLevelFromZstd(level int) EncoderLevel
		
			func WithEncoderLevel(l EncoderLevel) EOption
		
			const SpeedBestCompression
			const SpeedBetterCompression
			const SpeedDefault
			const SpeedFastest
			const github.com/parquet-go/parquet-go/compress/zstd.DefaultLevel
			const github.com/parquet-go/parquet-go/compress/zstd.SpeedBestCompression
			const github.com/parquet-go/parquet-go/compress/zstd.SpeedBetterCompression
			const github.com/parquet-go/parquet-go/compress/zstd.SpeedDefault
			const github.com/parquet-go/parquet-go/compress/zstd.SpeedFastest


	
		EOption is an option for creating a encoder.

		
			func WithAllLitEntropyCompression(b bool) EOption
			func WithEncoderConcurrency(n int) EOption
			func WithEncoderCRC(b bool) EOption
			func WithEncoderDict(dict []byte) EOption
			func WithEncoderDictRaw(id uint32, content []byte) EOption
			func WithEncoderLevel(l EncoderLevel) EOption
			func WithEncoderPadding(n int) EOption
			func WithLowerEncoderMem(b bool) EOption
			func WithNoEntropyCompression(b bool) EOption
			func WithSingleSegment(b bool) EOption
			func WithWindowSize(n int) EOption
			func WithZeroFrames(b bool) EOption
		
			func NewWriter(w io.Writer, opts ...EOption) (*Encoder, error)
			func ZipCompressor(opts ...EOption) func(w io.Writer) (io.WriteCloser, error)
			func github.com/hamba/avro/v2/ocf.WithZStandardEncoderOptions(opts ...EOption) ocf.EncoderFunc


	
		Header contains information about the first frame and block within that.

		
			
				Dictionary ID.
				If 0, no dictionary.

			
				First block information.

			
				FrameContentSize is the expected uncompressed size of the entire frame.

			
				If set there is a checksum present for the block content.
				The checksum field at the end is always 4 bytes long.

			
				HasFCS specifies whether FrameContentSize has a valid value.

			
				HeaderSize is the raw size of the frame header.
				
				For normal frames, it includes the size of the magic number and
				the size of the header (per section 3.1.1.1).
				It does not include the size for any data blocks (section 3.1.1.2) nor
				the size for the trailing content checksum.
				
				For skippable frames, this counts the size of the magic number
				along with the size of the size field of the payload.
				It does not include the size of the skippable payload itself.
				The total frame size is the HeaderSize plus the SkippableSize.

			
				SingleSegment specifies whether the data is to be decompressed into a
				single contiguous memory segment.
				It implies that WindowSize is invalid and that FrameContentSize is valid.

			
				Skippable will be true if the frame is meant to be skipped.
				This implies that FirstBlock.OK is false.

			
				SkippableID is the user-specific ID for the skippable frame.
				Valid values are between 0 to 15, inclusive.

			
				SkippableSize is the length of the user data to skip following
				the header.

			
				WindowSize is the window of data to keep while decoding.
				Will only be set if SingleSegment is false.

		
			
				AppendTo will append the encoded header to the dst slice.
				There is no error checking performed on the header values.

			
				Decode the header from the beginning of the stream.
				This will decode the frame header and the first block header if enough bytes are provided.
				It is recommended to provide at least HeaderMaxSize bytes.
				If the frame header cannot be read an error will be returned.
				If there isn't enough input, io.ErrUnexpectedEOF is returned.
				The FirstBlock.OK will indicate if enough information was available to decode the first block header.

			
				DecodeAndStrip will decode the header from the beginning of the stream
				and on success return the remaining bytes.
				This will decode the frame header and the first block header if enough bytes are provided.
				It is recommended to provide at least HeaderMaxSize bytes.
				If the frame header cannot be read an error will be returned.
				If there isn't enough input, io.ErrUnexpectedEOF is returned.
				The FirstBlock.OK will indicate if enough information was available to decode the first block header.



	
		SnappyConverter can read SnappyConverter-compressed streams and convert them to zstd.
		Conversion is done by converting the stream directly from Snappy without intermediate
		full decoding.
		Therefore the compression ratio is much less than what can be done by a full decompression
		and compression, and a faulty Snappy stream may lead to a faulty Zstandard stream without
		any errors being generated.
		No CRC value is being generated and not all CRC values of the Snappy stream are checked.
		However, it provides really fast recompression of Snappy streams.
		The converter can be reused to avoid allocations, even after errors.

		
			
				Convert the Snappy stream supplied in 'in' and write the zStandard stream to 'w'.
				If any error is detected on the Snappy stream it is returned.
				The number of bytes written is returned.





Package-Level Functions (total 29)


	
		EncoderLevelFromString will convert a string representation of an encoding level back
		to a compression level. The compare is not case sensitive.
		If the string wasn't recognized, (false, SpeedDefault) will be returned.


	
		EncoderLevelFromZstd will return an encoder level that closest matches the compression
		ratio of a specific zstd compression level.
		Many input values will provide the same compression level.


	
		IgnoreChecksum allows to forcibly ignore checksum checking.


	
		InspectDictionary loads a zstd dictionary and provides functions to inspect the content.


	
		NewReader creates a new decoder.
		A nil Reader can be provided in which case Reset can be used to start a decode.
		
		A Decoder can be used in two modes:
		
		1) As a stream, or
		2) For stateless decoding using DecodeAll.
		
		Only a single stream can be decoded concurrently, but the same decoder
		can run multiple concurrent stateless decodes. It is even possible to
		use stateless decodes while a stream is being decoded.
		
		The Reset function can be used to initiate a new stream, which will considerably
		reduce the allocations normally caused by NewReader.


	
		NewWriter will create a new Zstandard encoder.
		If the encoder will be used for encoding blocks a nil writer can be used.


	
		WithAllLitEntropyCompression will apply entropy compression if no matches are found.
		Disabling this will skip incompressible data faster, but in cases with no matches but
		skewed character distribution compression is lost.
		Default value depends on the compression level selected.


	
		WithDecodeAllCapLimit will limit DecodeAll to decoding cap(dst)-len(dst) bytes,
		or any size set in WithDecoderMaxMemory.
		This can be used to limit decoding to a specific maximum output size.
		Disabled by default.


	
		WithDecodeBuffersBelow will fully decode readers that have a
		`Bytes() []byte` and `Len() int` interface similar to bytes.Buffer.
		This typically uses less allocations but will have the full decompressed object in memory.
		Note that DecodeAllCapLimit will disable this, as well as giving a size of 0 or less.
		Default is 128KiB.


	
		WithDecoderConcurrency sets the number of created decoders.
		When decoding block with DecodeAll, this will limit the number
		of possible concurrently running decodes.
		When decoding streams, this will limit the number of
		inflight blocks.
		When decoding streams and setting maximum to 1,
		no async decoding will be done.
		When a value of 0 is provided GOMAXPROCS will be used.
		By default this will be set to 4 or GOMAXPROCS, whatever is lower.


	
		WithDecoderDictRaw registers a dictionary that may be used by the decoder.
		The slice content can be arbitrary data.


	
		WithDecoderDicts allows to register one or more dictionaries for the decoder.
		
		Each slice in dict must be in the [dictionary format] produced by
		"zstd --train" from the Zstandard reference implementation.
		
		If several dictionaries with the same ID are provided, the last one will be used.


	
		WithDecoderLowmem will set whether to use a lower amount of memory,
		but possibly have to allocate more while running.


	
		WithDecoderMaxMemory allows to set a maximum decoded size for in-memory
		non-streaming operations or maximum window size for streaming operations.
		This can be used to control memory usage of potentially hostile content.
		Maximum is 1 << 63 bytes. Default is 64GiB.


	
		WithDecoderMaxWindow allows to set a maximum window size for decodes.
		This allows rejecting packets that will cause big memory usage.
		The Decoder will likely allocate more memory based on the WithDecoderLowmem setting.
		If WithDecoderMaxMemory is set to a lower value, that will be used.
		Default is 512MB, Maximum is ~3.75 TB as per zstandard spec.


	
		WithEncoderConcurrency will set the concurrency,
		meaning the maximum number of encoders to run concurrently.
		The value supplied must be at least 1.
		For streams, setting a value of 1 will disable async compression.
		By default this will be set to GOMAXPROCS.


	
		WithEncoderCRC will add CRC value to output.
		Output will be 4 bytes larger.


	
		WithEncoderDict allows to register a dictionary that will be used for the encode.
		
		The slice dict must be in the [dictionary format] produced by
		"zstd --train" from the Zstandard reference implementation.
		
		The encoder *may* choose to use no dictionary instead for certain payloads.


	
		WithEncoderDictRaw registers a dictionary that may be used by the encoder.
		
		The slice content may contain arbitrary data. It will be used as an initial
		history.


	
		WithEncoderLevel specifies a predefined compression level.


	
		WithEncoderPadding will add padding to all output so the size will be a multiple of n.
		This can be used to obfuscate the exact output size or make blocks of a certain size.
		The contents will be a skippable frame, so it will be invisible by the decoder.
		n must be > 0 and <= 1GB, 1<<30 bytes.
		The padded area will be filled with data from crypto/rand.Reader.
		If `EncodeAll` is used with data already in the destination, the total size will be multiple of this.


	
		WithLowerEncoderMem will trade in some memory cases trade less memory usage for
		slower encoding speed.
		This will not change the window size which is the primary function for reducing
		memory usage. See WithWindowSize.


	
		WithNoEntropyCompression will always skip entropy compression of literals.
		This can be useful if content has matches, but unlikely to benefit from entropy
		compression. Usually the slight speed improvement is not worth enabling this.


	
		WithSingleSegment will set the "single segment" flag when EncodeAll is used.
		If this flag is set, data must be regenerated within a single continuous memory segment.
		In this case, Window_Descriptor byte is skipped, but Frame_Content_Size is necessarily present.
		As a consequence, the decoder must allocate a memory segment of size equal or larger than size of your content.
		In order to preserve the decoder from unreasonable memory requirements,
		a decoder is allowed to reject a compressed frame which requests a memory size beyond decoder's authorized range.
		For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB.
		This is only a recommendation, each decoder is free to support higher or lower limits, depending on local limitations.
		If this is not specified, block encodes will automatically choose this based on the input size and the window size.
		This setting has no effect on streamed encodes.


	
		WithWindowSize will set the maximum allowed back-reference distance.
		The value must be a power of two between MinWindowSize and MaxWindowSize.
		A larger value will enable better compression but allocate more memory and,
		for above-default values, take considerably longer.
		The default value is determined by the compression level and max 8MB.


	
		WithZeroFrames will encode 0 length input as full frames.
		This can be needed for compatibility with zstandard usage,
		but is not needed for this package.


	
		ZipCompressor returns a compressor that can be registered with zip libraries.
		The provided encoder options will be used on all encodes.


	
		ZipDecompressor returns a decompressor that can be registered with zip libraries.
		See ZipCompressor for example.
		Options can be specified. WithDecoderConcurrency(1) is forced,
		and by default a 128MB maximum decompression window is specified.
		The window size can be overridden if required.




Package-Level Variables (total 18)


	
		ErrBlockTooSmall is returned when a block is too small to be decoded.
		Typically returned on invalid input.


	
		ErrCompressedSizeTooBig is returned when a block is bigger than allowed.
		Typically this indicates wrong or corrupted input.


	
		ErrCRCMismatch is returned if CRC mismatches.


	
		ErrDecoderClosed will be returned if the Decoder was used after
		Close has been called.


	
		ErrDecoderNilInput is returned when a nil Reader was provided
		and an operation other than Reset/DecodeAll/Close was attempted.


	
		ErrDecoderSizeExceeded is returned if decompressed size exceeds the configured limit.


	
		ErrEncoderClosed will be returned if the Encoder was used after
		Close has been called.


	
		ErrFrameSizeExceeded is returned if the stated frame size is exceeded.
		This is only returned if SingleSegment is specified on the frame.


	
		ErrFrameSizeMismatch is returned if the stated frame size does not match the expected size.
		This is only returned if SingleSegment is specified on the frame.


	
		ErrMagicMismatch is returned when a "magic" number isn't what is expected.
		Typically this indicates wrong or corrupted input.


	
		ErrReservedBlockType is returned when a reserved block type is found.
		Typically this indicates wrong or corrupted input.


	
		ErrSnappyCorrupt reports that the input is invalid.


	
		ErrSnappyTooLarge reports that the uncompressed length is too large.


	
		ErrSnappyUnsupported reports that the input isn't supported.


	
		ErrUnexpectedBlockSize is returned when a block has unexpected size.
		Typically returned on invalid input.


	
		ErrUnknownDictionary is returned if the dictionary ID is unknown.


	
		ErrWindowSizeExceeded is returned when a reference exceeds the valid window size.
		Typically this indicates wrong or corrupted input.


	
		ErrWindowSizeTooSmall is returned when no window size is specified.
		Typically this indicates wrong or corrupted input.




Package-Level Constants (total 9)


	
		HeaderMaxSize is the maximum size of a Frame and Block Header.
		If less is sent to Header.Decode it *may* still contain enough information.


	
		MaxWindowSize is the maximum encoder window size
		and the default decoder maximum window size.


	
		MinWindowSize is the minimum Window Size, which is 1 KB.


	
		SpeedBestCompression will choose the best available compression option.
		This will offer the best compression no matter the CPU cost.


	
		SpeedBetterCompression will yield better compression than the default.
		Currently it is about zstd level 7-8 with ~ 2x-3x the default CPU usage.
		By using this, notice that CPU usage may go up in the future.


	
		SpeedDefault is the default "pretty fast" compression option.
		This is roughly equivalent to the default Zstandard mode (level 3).


	
		SpeedFastest will choose the fastest reasonable compression.
		This is roughly equivalent to the fastest Zstandard mode.


	
		ZipMethodPKWare is the original method number used by PKWARE to indicate Zstandard compression.
		Deprecated: This has been deprecated by PKWARE, use ZipMethodWinZip instead for compression.
		See https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.9.TXT


	
		ZipMethodWinZip is the method for Zstandard compressed data inside Zip files for WinZip.
		See https://www.winzip.com/win/en/comp_info.html