package types
Import Path
github.com/polarsignals/wal/types (on go.dev)
Dependency Relation
imports 3 packages, and imported by 5 packages
Package-Level Type Names (total 10)
LogEntry represents an entry that has already been encoded.
Data []byte
Index uint64
func SegmentReader.GetLog(idx uint64, le *LogEntry) error
func SegmentWriter.Append(entries []LogEntry) error
func SegmentWriter.GetLog(idx uint64, le *LogEntry) error
func github.com/polarsignals/wal.LogStore.GetLog(index uint64, log *LogEntry) error
func github.com/polarsignals/wal.LogStore.StoreLogs(logs []LogEntry) error
func github.com/polarsignals/wal.(*WAL).GetLog(index uint64, log *LogEntry) error
func github.com/polarsignals/wal.(*WAL).StoreLogs(encoded []LogEntry) error
func github.com/polarsignals/wal/segment.(*Reader).GetLog(idx uint64, le *LogEntry) error
func github.com/polarsignals/wal/segment.(*Writer).Append(entries []LogEntry) error
func github.com/polarsignals/wal/segment.(*Writer).GetLog(idx uint64, le *LogEntry) error
MetaStore is the interface we need to some persistent, crash safe backend. We
implement it with BoltDB for real usage but the interface allows alternatives
to be used, or tests to mock out FS access.
( MetaStore) Close() error
CommitState must atomically replace all persisted metadata in the current
store with the set provided. It must not return until the data is persisted
durably and in a crash-safe way otherwise the guarantees of the WAL will be
compromised. The WAL will only ever call this in a single thread at one
time and it will never be called concurrently with Load however it may be
called concurrently with Get/SetStable operations.
Load loads the existing persisted state. If there is no existing state
implementations are expected to create initialize new storage and return an
empty state.
*github.com/polarsignals/wal/metadb.BoltMetaDB
MetaStore : github.com/prometheus/common/expfmt.Closer
MetaStore : io.Closer
func github.com/polarsignals/wal.WithMetaStore(db MetaStore) wal.walOpt
PersistentState represents the WAL file metadata we need to store reliably to
recover on restart.
NextSegmentID uint64
Segments []SegmentInfo
func MetaStore.Load(dir string) (PersistentState, error)
func github.com/polarsignals/wal/metadb.(*BoltMetaDB).Load(dir string) (PersistentState, error)
func MetaStore.CommitState(PersistentState) error
func github.com/polarsignals/wal/metadb.(*BoltMetaDB).CommitState(state PersistentState) error
ReadableFile provides random read access to a file.
( ReadableFile) Close() error
( ReadableFile) ReadAt(p []byte, off int64) (n int, err error)
WritableFile (interface)
*github.com/polarsignals/wal/fs.File
*github.com/polarsignals/frostdb/storage.FileReaderAt
github.com/coreos/etcd/pkg/fileutil.LockedFile
github.com/ncruces/go-sqlite3/vfs.File (interface)
github.com/ncruces/go-sqlite3/vfs.FileBatchAtomicWrite (interface)
github.com/ncruces/go-sqlite3/vfs.FileBusyHandler (interface)
github.com/ncruces/go-sqlite3/vfs.FileCheckpoint (interface)
github.com/ncruces/go-sqlite3/vfs.FileChunkSize (interface)
github.com/ncruces/go-sqlite3/vfs.FileCommitPhaseTwo (interface)
github.com/ncruces/go-sqlite3/vfs.FileHasMoved (interface)
github.com/ncruces/go-sqlite3/vfs.FileLockState (interface)
github.com/ncruces/go-sqlite3/vfs.FileOverwrite (interface)
github.com/ncruces/go-sqlite3/vfs.FilePersistWAL (interface)
github.com/ncruces/go-sqlite3/vfs.FilePowersafeOverwrite (interface)
github.com/ncruces/go-sqlite3/vfs.FilePragma (interface)
github.com/ncruces/go-sqlite3/vfs.FileSharedMemory (interface)
github.com/ncruces/go-sqlite3/vfs.FileSizeHint (interface)
github.com/ncruces/go-sqlite3/vfs.FileSync (interface)
github.com/ncruces/go-sqlite3/vfs.FileUnwrap (interface)
mime/multipart.File (interface)
*os.File
ReadableFile : github.com/prometheus/common/expfmt.Closer
ReadableFile : io.Closer
ReadableFile : io.ReaderAt
func VFS.OpenReader(dir, name string) (ReadableFile, error)
func github.com/polarsignals/wal/fs.(*FS).OpenReader(dir string, name string) (ReadableFile, error)
SegmentFiler is the interface that provides access to segments to the WAL. It
encapsulated creating, and recovering segments and returning reader or writer
interfaces to interact with them. It's main purpose is to abstract the core
WAL logic both from the actual encoding layer of segment files. You can think
of it as a layer of abstraction above the VFS which abstracts actual file
system operations on files but knows nothing about the format. In tests for
example we can implement a SegmentFiler that is way simpler than the real
encoding/decoding layer on top of a VFS - even an in-memory VFS which makes
tests much simpler to write and run.
Create adds a new segment with the given info and returns a writer or an
error.
Delete removes the segment with given baseIndex and id if it exists. Note
that baseIndex is technically redundant since ID is unique on it's own. But
in practice we name files (or keys) with both so that they sort correctly.
This interface allows a simpler implementation where we can just delete
the file if it exists without having to scan the underlying storage for a.
List returns the set of segment IDs currently stored. It's used by the WAL
on recovery to find any segment files that need to be deleted following a
unclean shutdown. The returned map is a map of ID -> BaseIndex. BaseIndex
is returned to allow subsequent Delete calls to be made.
Open an already sealed segment for reading. Open may validate the file's
header and return an error if it doesn't match the expected info.
RecoverTail is called on an unsealed segment when re-opening the WAL it
will attempt to recover from a possible crash. It will either return an
error, or return a valid segmentWriter that is ready for further appends.
If the expected tail segment doesn't exist it must return an error wrapping
os.ErrNotExist.
*github.com/polarsignals/wal/segment.Filer
func github.com/polarsignals/wal.WithSegmentFiler(sf SegmentFiler) wal.walOpt
SegmentInfo is the metadata describing a single WAL segment.
BaseIndex is the raft index of the first entry that will be written to the
segment.
CreateTime records when the segment was first created.
ID uniquely identifies this segment file
IndexStart is the file offset where the index can be read from it's 0 for
tail segments and only set after a segment is sealed.
MaxIndex is the logical highest index that still exists in the segment. It
may be lower than the actual highest index if a tail truncation has
"deleted" a suffix of the segment. It is zero for unsealed segments and
only set one seal.
MinIndex is the logical lowest index that still exists in the segment. It
may be greater than BaseIndex if a head truncation has "deleted" a prefix
of the segment.
SealTime records when the segment was sealed. Zero indicates that it's not
sealed yet.
SizeLimit is the soft limit for the segment's size. The segment file may be
pre-allocated to this size on filesystems that support it. It is a soft
limit in the sense that the final Append usually takes the segment file
past this size before it is considered full and sealed.
func SegmentFiler.Create(info SegmentInfo) (SegmentWriter, error)
func SegmentFiler.Open(info SegmentInfo) (SegmentReader, error)
func SegmentFiler.RecoverTail(info SegmentInfo) (SegmentWriter, error)
func github.com/polarsignals/wal/segment.FileName(i SegmentInfo) string
func github.com/polarsignals/wal/segment.(*Filer).Create(info SegmentInfo) (SegmentWriter, error)
func github.com/polarsignals/wal/segment.(*Filer).Open(info SegmentInfo) (SegmentReader, error)
func github.com/polarsignals/wal/segment.(*Filer).RecoverTail(info SegmentInfo) (SegmentWriter, error)
SegmentReader wraps a ReadableFile to allow lookup of logs in an existing
segment file. It's an interface to make testing core WAL simpler. The first
call will always be validate which passes in the ReaderAt to be used for
subsequent reads.
( SegmentReader) Close() error
GetLog writes the raw log entry bytes associated with idx into le.Data.
If the log doesn't exist in this segment ErrNotFound must be returned.
SegmentWriter (interface)
github.com/polarsignals/wal.LogStore (interface)
*github.com/polarsignals/wal.WAL
*github.com/polarsignals/wal/segment.Reader
*github.com/polarsignals/wal/segment.Writer
SegmentReader : github.com/prometheus/common/expfmt.Closer
SegmentReader : io.Closer
func SegmentFiler.Open(info SegmentInfo) (SegmentReader, error)
func github.com/polarsignals/wal/segment.(*Filer).Open(info SegmentInfo) (SegmentReader, error)
SegmentWriter manages appending logs to the tail segment of the WAL. It's an
interface to make testing core WAL simpler. Every SegmentWriter will have
either `init` or `recover` called once before any other methods. When either
returns it must either return an error or be ready to accept new writes and
reads.
Append adds one or more entries. It must not return until the entries are
durably stored otherwise raft's guarantees will be compromised. Append must
not be called concurrently with any other call to Sealed or Append.
( SegmentWriter) Close() error
GetLog writes the raw log entry bytes associated with idx into le.Data.
If the log doesn't exist in this segment ErrNotFound must be returned.
LastIndex returns the most recently persisted index in the log. It must
respond without blocking on Append since it's needed frequently by read
paths that may call it concurrently. Typically this will be loaded from an
atomic int. If the segment is empty lastIndex should return zero.
Sealed returns whether the segment is sealed or not. If it is it returns
true and the file offset that it's index array starts at to be saved in
meta data. WAL will call this after every append so it should be relatively
cheap in the common case. This design allows the final Append to write out
the index or any additional data needed at seal time in the same fsync.
Sealed must not be called concurrently with any other call to Sealed or
Append.
*github.com/polarsignals/wal/segment.Writer
SegmentWriter : SegmentReader
SegmentWriter : github.com/prometheus/common/expfmt.Closer
SegmentWriter : io.Closer
func SegmentFiler.Create(info SegmentInfo) (SegmentWriter, error)
func SegmentFiler.RecoverTail(info SegmentInfo) (SegmentWriter, error)
func github.com/polarsignals/wal/segment.(*Filer).Create(info SegmentInfo) (SegmentWriter, error)
func github.com/polarsignals/wal/segment.(*Filer).RecoverTail(info SegmentInfo) (SegmentWriter, error)
VFS is the interface WAL needs to interact with the file system. In
production it would normally be implemented by RealFS which interacts with
the operating system FS using standard go os package. It's useful to allow
testing both to run quicker (by being in memory only) and to make it easy to
simulate all kinds of disk errors and failure modes without needing a more
elaborate external test harness like ALICE.
Create creates a new file with the given name. If a file with the same name
already exists an error is returned. If a non-zero size is given,
implementations should make a best effort to pre-allocate the file to be
that size. The dir must already exist and be writable to the current
process.
Delete indicates the file is no longer required. Typically it should be
deleted from the underlying system to free disk space.
ListDir returns a list of all files in the specified dir in lexicographical
order. If the dir doesn't exist, it must return an error. Empty array with
nil error is assumed to mean that the directory exists and was readable,
but contains no files.
OpenReader opens an existing file in read-only mode. If the file doesn't
exist or permission is denied, an error is returned, otherwise no checks
are made about the well-formedness of the file, it may be empty, the wrong
size or corrupt in arbitrary ways.
OpenWriter opens a file in read-write mode. If the file doesn't exist or
permission is denied, an error is returned, otherwise no checks are made
about the well-formedness of the file, it may be empty, the wrong size or
corrupt in arbitrary ways.
*github.com/polarsignals/wal/fs.FS
func github.com/polarsignals/wal/segment.NewFiler(dir string, vfs VFS) *segment.Filer
WritableFile provides random read-write access to a file as well as the
ability to fsync it to disk.
( WritableFile) Close() error
( WritableFile) ReadAt(p []byte, off int64) (n int, err error)
( WritableFile) Sync() error
( WritableFile) WriteAt(p []byte, off int64) (n int, err error)
*github.com/polarsignals/wal/fs.File
github.com/coreos/etcd/pkg/fileutil.LockedFile
*os.File
WritableFile : ReadableFile
WritableFile : github.com/polarsignals/frostdb.Sync
WritableFile : github.com/prometheus/common/expfmt.Closer
WritableFile : io.Closer
WritableFile : io.ReaderAt
WritableFile : io.WriterAt
func VFS.Create(dir, name string, size uint64) (WritableFile, error)
func VFS.OpenWriter(dir, name string) (WritableFile, error)
func github.com/polarsignals/wal/fs.(*FS).Create(dir string, name string, size uint64) (WritableFile, error)
func github.com/polarsignals/wal/fs.(*FS).OpenWriter(dir string, name string) (WritableFile, error)
 |
The pages are generated with Golds v0.8.2. (GOOS=linux GOARCH=amd64) Golds is a Go 101 project developed by Tapir Liu. PR and bug reports are welcome and can be submitted to the issue list. Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds. |