package z

Import Path
	github.com/dgraph-io/ristretto/v2/z (on go.dev)

Dependency Relation
	imports 26 packages, and imported by 5 packages

Involved Source Files

	    allocator.go
	    bbloom.go
	    btree.go
	    buffer.go
	    calloc.go
	    calloc_64bit.go
	    calloc_nojemalloc.go
	    file.go
	    file_linux.go
	    flags.go
	    histogram.go
	    mmap.go
	    mmap_linux.go
	    mremap_size.go
	    rtutil.go
	    z.go
	    rtutil.s
Package-Level Type Names (total 15)

	/* sort by: alphabet | popularity */
	 type Allocator (struct)
		Allocator amortizes the cost of small allocations by allocating memory in
		bigger chunks.  Internally it uses z.Calloc to allocate memory. Once
		allocated, the memory is not moved, so it is safe to use the allocated bytes
		to unsafe cast them to Go struct pointers. Maintaining a freelist is slow.
		Instead, Allocator only allocates memory, with the idea that finally we
		would just release the entire Allocator.

		Fields (total 3)
			Mutex sync.Mutex
			Ref uint64
			Tag string
		Methods (total 13)
			(*Allocator) Allocate(sz int) []byte
			(*Allocator) AllocateAligned(sz int) []byte
			(*Allocator) Allocated() uint64
			(*Allocator) Copy(buf []byte) []byte
			(*Allocator) Lock()
				Lock locks m.
				If the lock is already in use, the calling goroutine
				blocks until the mutex is available.

			(*Allocator) MaxAlloc() int
			(*Allocator) Release()
				Release would release the memory back. Remember to make this call to avoid memory leaks.

			(*Allocator) Reset()
			(*Allocator) Size() int
				Size returns the size of the allocations so far.

			(*Allocator) String() string
			(*Allocator) TrimTo(max int)
			(*Allocator) TryLock() bool
				TryLock tries to lock m and reports whether it succeeded.
				
				Note that while correct uses of TryLock do exist, they are rare,
				and use of TryLock is often a sign of a deeper problem
				in a particular use of mutexes.

			(*Allocator) Unlock()
				Unlock unlocks m.
				It is a run-time error if m is not locked on entry to Unlock.
				
				A locked [Mutex] is not associated with a particular goroutine.
				It is allowed for one goroutine to lock a Mutex and then
				arrange for another goroutine to unlock it.

		Implements (at least 4)
			*Allocator : github.com/gogo/protobuf/proto.Sizer
			*Allocator : expvar.Var
			*Allocator : fmt.Stringer
			*Allocator : sync.Locker
		As Outputs Of (at least 3)
			func AllocatorFrom(ref uint64) *Allocator
			func NewAllocator(sz int, tag string) *Allocator
			func (*AllocatorPool).Get(sz int, tag string) *Allocator
		As Inputs Of (at least 2)
			func (*AllocatorPool).Return(a *Allocator)
			func github.com/dgraph-io/badger/v4/y.NewKV(alloc *Allocator) *pb.KV

	 type AllocatorPool (struct)

		Methods (total 3)
			(*AllocatorPool) Get(sz int, tag string) *Allocator
			(*AllocatorPool) Release()
			(*AllocatorPool) Return(a *Allocator)
		As Outputs Of (at least one exported)
			func NewAllocatorPool(sz int) *AllocatorPool

	 type Bloom (struct)
		Bloom filter

		Fields (only one)
			ElemNum uint64
		Methods (total 9)
			(*Bloom) Add(hash uint64)
				Add adds hash of a key to the bloomfilter.

			(*Bloom) AddIfNotHas(hash uint64) bool
				AddIfNotHas only Adds hash, if it's not present in the bloomfilter.
				Returns true if hash was added.
				Returns false if hash was already registered in the bloomfilter.

			(*Bloom) Clear()
				Clear resets the Bloom filter.

			( Bloom) Has(hash uint64) bool
				Has checks if bit(s) for entry hash is/are set,
				returns true if the hash was added to the Bloom Filter.

			(*Bloom) IsSet(idx uint64) bool
				IsSet checks if bit[idx] of bitset is set, returns true/false.

			( Bloom) JSONMarshal() []byte
				JSONMarshal returns JSON-object (type bloomJSONImExport) as []byte.

			(*Bloom) Set(idx uint64)
				Set sets the bit[idx] of bitset.

			(*Bloom) Size(sz uint64)
				Size makes Bloom filter with as bitset of size sz.

			(*Bloom) TotalSize() int
				TotalSize returns the total size of the bloom filter.

		As Outputs Of (at least 2)
			func JSONUnmarshal(dbData []byte) (*Bloom, error)
			func NewBloomFilter(params ...float64) (bloomfilter *Bloom)

	 type Buffer (struct)
		Buffer is equivalent of bytes.Buffer without the ability to read. It is NOT thread-safe.
		
		In UseCalloc mode, z.Calloc is used to allocate memory, which depending upon how the code is
		compiled could use jemalloc for allocations.
		
		In UseMmap mode, Buffer  uses file mmap to allocate memory. This allows us to store big data
		structures without using physical memory.
		
		MaxSize can be set to limit the memory usage.

		Methods (total 21)
			(*Buffer) Allocate(n int) []byte
				Allocate is a way to get a slice of size n back from the buffer. This slice can be directly
				written to. Warning: Allocate is not thread-safe. The byte slice returned MUST be used before
				further calls to Buffer.

			(*Buffer) AllocateOffset(n int) int
				AllocateOffset works the same way as allocate, but instead of returning a byte slice, it returns
				the offset of the allocation.

			(*Buffer) Bytes() []byte
				Bytes would return all the written bytes as a slice.

			(*Buffer) Data(offset int) []byte
			(*Buffer) Grow(n int)
				Grow would grow the buffer to have at least n more bytes. In case the buffer is at capacity, it
				would reallocate twice the size of current capacity + n, to ensure n bytes can be written to the
				buffer without further allocation. In UseMmap mode, this might result in underlying file
				expansion.

			(*Buffer) IsEmpty() bool
			(*Buffer) LenNoPadding() int
				LenNoPadding would return the number of bytes written to the buffer so far
				(without the padding).

			(*Buffer) LenWithPadding() int
				LenWithPadding would return the number of bytes written to the buffer so far
				plus the padding at the start of the buffer.

			(*Buffer) Release() error
				Release would free up the memory allocated by the buffer. Once the usage of buffer is done, it is
				important to call Release, otherwise a memory leak can happen.

			(*Buffer) Reset()
				Reset would reset the buffer to be reused.

			(*Buffer) Slice(offset int) ([]byte, int)
				Slice would return the slice written at offset.

			(*Buffer) SliceAllocate(sz int) []byte
				SliceAllocate would encode the size provided into the buffer, followed by a call to Allocate,
				hence returning the slice of size sz. This can be used to allocate a lot of small buffers into
				this big buffer.
				Note that SliceAllocate should NOT be mixed with normal calls to Write.

			(*Buffer) SliceIterate(f func(slice []byte) error) error
			(*Buffer) SliceOffsets() []int
				SliceOffsets is an expensive function. Use sparingly.

			(*Buffer) SortSlice(less func(left, right []byte) bool)
				SortSlice is like SortSliceBetween but sorting over the entire buffer.

			(*Buffer) SortSliceBetween(start, end int, less LessFunc)
			(*Buffer) StartOffset() int
			(*Buffer) WithAutoMmap(threshold int, path string) *Buffer
			(*Buffer) WithMaxSize(size int) *Buffer
			(*Buffer) Write(p []byte) (n int, err error)
				Write would write p bytes to the buffer.

			(*Buffer) WriteSlice(slice []byte)
		Implements (at least 4)
			*Buffer : github.com/apache/arrow-go/v18/internal/hashing.ByteSlice
			*Buffer : github.com/miekg/dns.Writer
			*Buffer : internal/bisect.Writer
			*Buffer : io.Writer
		As Outputs Of (at least 6)
			func NewBuffer(capacity int, tag string) *Buffer
			func NewBufferPersistent(path string, capacity int) (*Buffer, error)
			func NewBufferSlice(slice []byte) *Buffer
			func NewBufferTmp(dir string, capacity int) (*Buffer, error)
			func (*Buffer).WithAutoMmap(threshold int, path string) *Buffer
			func (*Buffer).WithMaxSize(size int) *Buffer
		As Inputs Of (at least 4)
			func github.com/dgraph-io/badger/v4.BufferToKVList(buf *Buffer) (*pb.KVList, error)
			func github.com/dgraph-io/badger/v4.KVToBuffer(kv *pb.KV, buf *Buffer)
			func github.com/dgraph-io/badger/v4.(*StreamWriter).Write(buf *Buffer) error
			func github.com/dgraph-io/badger/v4.(*WriteBatch).Write(buf *Buffer) error

	 type BufferType int (basic type)

		Methods (only one)
			( BufferType) String() string
		Implements (at least 2)
			 BufferType : expvar.Var
			 BufferType : fmt.Stringer
		As Types Of (total 3)
			const UseCalloc
			const UseInvalid
			const UseMmap

	 type Closer (struct)
		Closer holds the two things we need to close a goroutine and wait for it to
		finish: a chan to tell the goroutine to shut down, and a WaitGroup with
		which to wait for it to finish shutting down.

		Methods (total 7)
			(*Closer) AddRunning(delta int)
				AddRunning Add()'s delta to the WaitGroup.

			(*Closer) Ctx() context.Context
				Ctx can be used to get a context, which would automatically get cancelled when Signal is called.

			(*Closer) Done()
				Done calls Done() on the WaitGroup.

			(*Closer) HasBeenClosed() <-chan struct{}
				HasBeenClosed gets signaled when Signal() is called.

			(*Closer) Signal()
				Signal signals the HasBeenClosed signal.

			(*Closer) SignalAndWait()
				SignalAndWait calls Signal(), then Wait().

			(*Closer) Wait()
				Wait waits on the WaitGroup. (It waits for NewCloser's initial value, AddRunning, and Done
				calls to balance out.)

		As Outputs Of (at least one exported)
			func NewCloser(initial int) *Closer
		As Inputs Of (at least one exported)
			func github.com/dgraph-io/badger/v4/y.(*WaterMark).Init(closer *Closer)

	 type HistogramData (struct)
		HistogramData stores the information needed to represent the sizes of the keys and values
		as a histogram.

		Fields (total 6)
			Bounds []float64
			Count int64
			CountPerBucket []int64
			Max int64
			Min int64
			Sum int64
		Methods (total 6)
			(*HistogramData) Clear()
				Clear reset the histogram. Helpful in situations where we need to reset the metrics

			(*HistogramData) Copy() *HistogramData
			(*HistogramData) Mean() float64
				Mean returns the mean value for the histogram.

			(*HistogramData) Percentile(p float64) float64
				Percentile returns the percentile value for the histogram.
				value of p should be between [0.0-1.0]

			(*HistogramData) String() string
				String converts the histogram data into human-readable string.

			(*HistogramData) Update(value int64)
				Update changes the Min and Max fields if value is less than or greater than the current values.

		Implements (at least 2)
			*HistogramData : expvar.Var
			*HistogramData : fmt.Stringer
		As Outputs Of (at least 3)
			func NewHistogramData(bounds []float64) *HistogramData
			func (*HistogramData).Copy() *HistogramData
			func github.com/dgraph-io/ristretto/v2.(*Metrics).LifeExpectancySeconds() *HistogramData

	 type Key (interface)
	 type LessFunc (func)

		As Inputs Of (at least one exported)
			func (*Buffer).SortSliceBetween(start, end int, less LessFunc)

	 type MemStats (struct)
		MemStats is used to fetch JE Malloc Stats. The stats are fetched from
		the mallctl namespace http://jemalloc.net/jemalloc.3.html#mallctl_namespace.

		Fields (total 4)
			Active uint64
				Total number of bytes in active pages allocated by the application. This
				is a multiple of the page size, and greater than or equal to
				Allocated.
				http://jemalloc.net/jemalloc.3.html#stats.active

			Allocated uint64
				Total number of bytes allocated by the application.
				http://jemalloc.net/jemalloc.3.html#stats.allocated

			Resident uint64
				Maximum number of bytes in physically resident data pages mapped by the
				allocator, comprising all pages dedicated to allocator metadata, pages
				backing active allocations, and unused dirty pages. This is a maximum
				rather than precise because pages may not actually be physically
				resident if they correspond to demand-zeroed virtual memory that has not
				yet been touched. This is a multiple of the page size, and is larger
				than stats.active.
				http://jemalloc.net/jemalloc.3.html#stats.resident

			Retained uint64
				Total number of bytes in virtual memory mappings that were retained
				rather than being returned to the operating system via e.g. munmap(2) or
				similar. Retained virtual memory is typically untouched, decommitted, or
				purged, so it has no strongly associated physical memory (see extent
				hooks http://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks for
				details). Retained memory is excluded from mapped memory statistics,
				e.g. stats.mapped (http://jemalloc.net/jemalloc.3.html#stats.mapped).
				http://jemalloc.net/jemalloc.3.html#stats.retained

		As Inputs Of (at least one exported)
			func ReadMemStats(_ *MemStats)

	 type MmapFile (struct)
		MmapFile represents an mmapd file and includes both the buffer to the data
		and the file descriptor.

		Fields (total 2)
			Data []byte
			Fd *os.File
		Methods (total 8)
			(*MmapFile) AllocateSlice(sz, offset int) ([]byte, int, error)
				AllocateSlice allocates a slice of the given size at the given offset.

			(*MmapFile) Bytes(off, sz int) ([]byte, error)
				Bytes returns data starting from offset off of size sz. If there's not enough data, it would
				return nil slice and io.EOF.

			(*MmapFile) Close(maxSz int64) error
				Close would close the file. It would also truncate the file if maxSz >= 0.

			(*MmapFile) Delete() error
			(*MmapFile) NewReader(offset int) io.Reader
			(*MmapFile) Slice(offset int) []byte
				Slice returns the slice at the given offset.

			(*MmapFile) Sync() error
			(*MmapFile) Truncate(maxSz int64) error
				Truncate would truncate the mmapped file to the given size. On Linux, we truncate
				the underlying file and then call mremap, but on other systems, we unmap first,
				then truncate, then re-map.

		Implements (at least one exported)
			*MmapFile : github.com/polarsignals/frostdb.Sync
		As Outputs Of (at least 2)
			func OpenMmapFile(filename string, flag int, maxSz int) (*MmapFile, error)
			func OpenMmapFileUsing(fd *os.File, sz int, writable bool) (*MmapFile, error)
		As Inputs Of (at least one exported)
			func github.com/dgraph-io/badger/v4/table.OpenTable(mf *MmapFile, opts table.Options) (*table.Table, error)

	 type SuperFlag (struct)

		Methods (total 11)
			(*SuperFlag) GetBool(opt string) bool
			(*SuperFlag) GetDuration(opt string) time.Duration
			(*SuperFlag) GetFloat64(opt string) float64
			(*SuperFlag) GetInt64(opt string) int64
			(*SuperFlag) GetPath(opt string) string
			(*SuperFlag) GetString(opt string) string
			(*SuperFlag) GetUint32(opt string) uint32
			(*SuperFlag) GetUint64(opt string) uint64
			(*SuperFlag) Has(opt string) bool
			(*SuperFlag) MergeAndCheckDefault(flag string) *SuperFlag
			(*SuperFlag) String() string
		Implements (at least 2)
			*SuperFlag : expvar.Var
			*SuperFlag : fmt.Stringer
		As Outputs Of (at least 2)
			func NewSuperFlag(flag string) *SuperFlag
			func (*SuperFlag).MergeAndCheckDefault(flag string) *SuperFlag

	 type SuperFlagHelp (struct)
		SuperFlagHelp makes it really easy to generate command line `--help` output for a SuperFlag. For
		example:
		
			const flagDefaults = `enabled=true; path=some/path;`
		
			var help string = z.NewSuperFlagHelp(flagDefaults).
				Flag("enabled", "Turns on <something>.").
				Flag("path", "The path to <something>.").
				Flag("another", "Not present in defaults, but still included.").
				String()
		
		The `help` string would then contain:
		
			enabled=true; Turns on <something>.
			path=some/path; The path to <something>.
			another=; Not present in defaults, but still included.
		
		All flags are sorted alphabetically for consistent `--help` output. Flags with default values are
		placed at the top, and everything else goes under.

		Methods (total 3)
			(*SuperFlagHelp) Flag(name, description string) *SuperFlagHelp
			(*SuperFlagHelp) Head(head string) *SuperFlagHelp
			(*SuperFlagHelp) String() string
		Implements (at least 2)
			*SuperFlagHelp : expvar.Var
			*SuperFlagHelp : fmt.Stringer
		As Outputs Of (at least 3)
			func NewSuperFlagHelp(defaults string) *SuperFlagHelp
			func (*SuperFlagHelp).Flag(name, description string) *SuperFlagHelp
			func (*SuperFlagHelp).Head(head string) *SuperFlagHelp

	 type Tree (struct)
		Tree represents the structure for custom mmaped B+ tree.
		It supports keys in range [1, math.MaxUint64-1] and values [1, math.Uint64].

		Methods (total 9)
			(*Tree) Close() error
				Close releases the memory used by the tree.

			(*Tree) DeleteBelow(ts uint64)
				DeleteBelow deletes all keys with value under ts.

			(*Tree) Get(k uint64) uint64
				Get looks for key and returns the corresponding value.
				If key is not found, 0 is returned.

			(*Tree) Iterate(fn func(node))
				Iterate iterates over the tree and executes the fn on each node.

			(*Tree) IterateKV(f func(key, val uint64) (newVal uint64))
				IterateKV iterates through all keys and values in the tree.
				If newVal is non-zero, it will be set in the tree.

			(*Tree) Print()
				Print iterates over the tree and prints all valid KVs.

			(*Tree) Reset()
				Reset resets the tree and truncates it to maxSz.

			(*Tree) Set(k, v uint64)
				Set sets the key-value pair in the tree.

			(*Tree) Stats() TreeStats
				Stats returns stats about the tree.

		Implements (at least 2)
			*Tree : github.com/prometheus/common/expfmt.Closer
			*Tree : io.Closer
		As Outputs Of (at least 2)
			func NewTree(tag string) *Tree
			func NewTreePersistent(path string) (*Tree, error)

	 type TreeStats (struct)

		Fields (total 7)
			Allocated int
				// Derived.

			Bytes int
				// Derived.

			NumLeafKeys int
				// Calculated.

			NumPages int
				// Derived.

			NumPagesFree int
				// Calculated.

			Occupancy float64
				// Derived.

			PageSize int
				// Derived.

		As Outputs Of (at least one exported)
			func (*Tree).Stats() TreeStats


Package-Level Functions (total 42)

	 func AllocatorFrom(ref uint64) *Allocator
		AllocatorFrom would return the allocator corresponding to the ref.

	 func Allocators() string
	 func BytesToUint64Slice(b []byte) []uint64
		BytesToUint64Slice converts a byte slice to a uint64 slice.

	 func Calloc(n int, tag string) []byte
		Calloc allocates a slice of size n.

	 func CallocNoRef(n int, tag string) []byte
		CallocNoRef will not give you memory back without jemalloc.

	 func CPUTicks() int64
		CPUTicks is a faster alternative to NanoTime to measure time duration.

	 func FastRand() uint32
		FastRand is a fast thread local random function.

	 func Fibonacci(num int) []float64
	 func Free(b []byte)
		Free does not do anything in this mode.

	 func HistogramBounds(minExponent, maxExponent uint32) []float64
		Creates bounds for an histogram. The bounds are powers of two of the form
		[2^min_exponent, ..., 2^max_exponent].

	 func JSONUnmarshal(dbData []byte) (*Bloom, error)
		JSONUnmarshal takes JSON-Object (type bloomJSONImExport) as []bytes
		returns bloom32 / bloom64 object.

	 func KeyToHash[K](key K) (uint64, uint64)

		Type Parameters:
			K: Key

		TODO: Figure out a way to re-use memhash for the second uint64 hash,
		we already know that appending bytes isn't reliable for generating a
		second hash (see Ristretto PR #88).
		We also know that while the Go runtime has a runtime memhash128
		function, it's not possible to use it to generate [2]uint64 or
		anything resembling a 128bit hash, even though that's exactly what
		we need in this situation.

	 func Leaks() string
	 func Madvise(b []byte, readahead bool) error
		Madvise uses the madvise system call to give advise about the use of memory
		when using a slice that is memory-mapped to a file. Set the readahead flag to
		false if page references are expected in random order.

	 func Memclr(b []byte)
	 func MemHash(data []byte) uint64
		MemHash is the hash function used by go map, it utilizes available hardware instructions(behaves
		as aeshash if aes instruction is available).
		NOTE: The hash seed changes for every process. So, this cannot be used as a persistent hash.

	 func MemHashString(str string) uint64
		MemHashString is the hash function used by go map, it utilizes available hardware instructions
		(behaves as aeshash if aes instruction is available).
		NOTE: The hash seed changes for every process. So, this cannot be used as a persistent hash.

	 func Mmap(fd *os.File, writable bool, size int64) ([]byte, error)
		Mmap uses the mmap system call to memory-map a file. If writable is true,
		memory protection of the pages is set so that they may be written to as well.

	 func Msync(b []byte) error
		Msync would call sync on the mmapped data.

	 func Munmap(b []byte) error
		Munmap unmaps a previously mapped slice.

	 func NanoTime() int64
		NanoTime returns the current time in nanoseconds from a monotonic clock.

	 func NewAllocator(sz int, tag string) *Allocator
		NewAllocator creates an allocator starting with the given size.

	 func NewAllocatorPool(sz int) *AllocatorPool
	 func NewBloomFilter(params ...float64) (bloomfilter *Bloom)
		NewBloomFilter returns a new bloomfilter.

	 func NewBuffer(capacity int, tag string) *Buffer
	 func NewBufferPersistent(path string, capacity int) (*Buffer, error)
		It is the caller's responsibility to set offset after this, because Buffer
		doesn't remember what it was.

	 func NewBufferSlice(slice []byte) *Buffer
	 func NewBufferTmp(dir string, capacity int) (*Buffer, error)
	 func NewCloser(initial int) *Closer
		NewCloser constructs a new Closer, with an initial count on the WaitGroup.

	 func NewHistogramData(bounds []float64) *HistogramData
		NewHistogramData returns a new instance of HistogramData with properly initialized fields.

	 func NewSuperFlag(flag string) *SuperFlag
	 func NewSuperFlagHelp(defaults string) *SuperFlagHelp
	 func NewTree(tag string) *Tree
		NewTree returns an in-memory B+ tree.

	 func NewTreePersistent(path string) (*Tree, error)
		NewTree returns a persistent on-disk B+ tree.

	 func NumAllocBytes() int64
		NumAllocBytes returns the number of bytes allocated using calls to z.Calloc. The allocations
		could be happening via either Go or jemalloc, depending upon the build flags.

	 func OpenMmapFile(filename string, flag int, maxSz int) (*MmapFile, error)
		OpenMmapFile opens an existing file or creates a new file. If the file is
		created, it would truncate the file to maxSz. In both cases, it would mmap
		the file to maxSz and returned it. In case the file is created, z.NewFile is
		returned.

	 func OpenMmapFileUsing(fd *os.File, sz int, writable bool) (*MmapFile, error)
	 func ReadMemStats(_ *MemStats)
		ReadMemStats doesn't do anything since all the memory is being managed
		by the Go runtime.

	 func SetTmpDir(dir string)
		SetTmpDir sets the temporary directory for the temporary buffers.

	 func StatsPrint()
	 func SyncDir(dir string) error
	 func ZeroOut(dst []byte, start, end int)
		ZeroOut zeroes out all the bytes in the range [start, end).


Package-Level Variables (only one)

	  var NewFile error
Package-Level Constants (total 5)

	const MaxArrayLen = 1125899906842623
		MaxArrayLen is a safe maximum length for slices on this architecture.

	const MaxBufferSize = 274877906944
		MaxBufferSize is the size of virtually unlimited buffer on this architecture.

	const UseCalloc BufferType = 0
	const UseInvalid BufferType = 2
	const UseMmap BufferType = 1

The pages are generated with Golds v0.8.4. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.