Compact will compact the given parts into a Parquet file written to the next level file.(*FileCompaction) MaxSize() int64 Reset is called when the level no longer has active parts in it at the end of a compaction. Snapshot takes a snapshot of the current level. It ignores the parts and just hard links the files into the snapshot directory.
It will rotate the active file if it has data in it rendering all snapshotted files as immutable. Sync calls Sync on the underlying file.
*FileCompaction : Level
*FileCompaction : github.com/polarsignals/frostdb.Sync
func NewFileCompaction(dir string, maxSize int64, compact Compaction, logger log.Logger) (*FileCompaction, error)
LSM is a log-structured merge-tree like index. It is implemented as a single linked list of parts.
Arrow records are always added to the L0 list. When a list reaches it's configured max size it is compacted
calling the levels Compact function and is then added as a new part to the next level.
[L0]->[record]->[record]->[L1]->[record/parquet]->[record/parquet] etc.RWMutexsync.RWMutex(*LSM) Add(tx uint64, record arrow.Record)(*LSM) Close() error EnsureCompaction forces a compaction of all levels, regardless of whether the
levels are below the target size. InsertPart inserts a part into the LSM tree. It will be inserted into the correct level. It does not check if the insert should cause a compaction.
This should only be used during snapshot recovery. It will drop the insert on the floor if the part is older than a part in the next level of the LSM. This indicates
that this part is already accounted for in the next level vis compaction.(*LSM) Iterate(iter func(node *Node) bool) LevelSize returns the size of a specific level in bytes. Lock locks rw for writing.
If the lock is already locked for reading or writing,
Lock blocks until the lock is available.(*LSM) MaxLevel() SentinelType(*LSM) Prefixes(_ context.Context, _ string) ([]string, error) RLock locks rw for reading.
It should not be used for recursive read locking; a blocked Lock
call excludes new readers from acquiring the lock. See the
documentation on the [RWMutex] type. RLocker returns a [Locker] interface that implements
the [Locker.Lock] and [Locker.Unlock] methods by calling rw.RLock and rw.RUnlock. RUnlock undoes a single [RWMutex.RLock] call;
it does not affect other simultaneous readers.
It is a run-time error if rw is not locked for reading
on entry to RUnlock. Rotate will write all parts in the LSM into the external writer. No changes are made to the LSM.(*LSM) Scan(ctx context.Context, _ string, _ *dynparquet.Schema, filter logicalplan.Expr, tx uint64, callback func(context.Context, any) error) error Size returns the total size of the index in bytes. Snapshot creates a snapshot of the index at the given transaction. It will call the writer function with the parts in the index that are in-memory.(*LSM) String() string TryLock tries to lock rw for writing and reports whether it succeeded.
Note that while correct uses of TryLock do exist, they are rare,
and use of TryLock is often a sign of a deeper problem
in a particular use of mutexes. TryRLock tries to lock rw for reading and reports whether it succeeded.
Note that while correct uses of TryRLock do exist, they are rare,
and use of TryRLock is often a sign of a deeper problem
in a particular use of mutexes. Unlock unlocks rw for writing. It is a run-time error if rw is
not locked for writing on entry to Unlock.
As with Mutexes, a locked [RWMutex] is not associated with a particular
goroutine. One goroutine may [RWMutex.RLock] ([RWMutex.Lock]) a RWMutex and then
arrange for another goroutine to [RWMutex.RUnlock] ([RWMutex.Unlock]) it.(*LSM) WaitForPendingCompactions()
*LSM : github.com/polarsignals/frostdb.DataSource
*LSM : github.com/prometheus/common/expfmt.Closer
*LSM : expvar.Var
*LSM : fmt.Stringer
*LSM : io.Closer
*LSM : sync.Locker
func NewLSM(dir string, schema *dynparquet.Schema, levels []*LevelConfig, watermark func() uint64, options ...LSMOption) (*LSM, error)
func github.com/polarsignals/frostdb.(*TableBlock).Index() *LSM
Node is a Part that is a part of a linked-list. Insert a Node into the list, in order by Tx. Iterate accesses every node in the list.(*Node) Part() parts.Part Prepend a node onto the front of the list. Sentinel adds a new sentinel node to the list, and returns the sub list starting from that sentinel.(*Node) String() string
*Node : expvar.Var
*Node : fmt.Stringer
func NewList(sentinel SentinelType) *Node
func (*Node).Prepend(part parts.Part) *Node
func (*Node).Sentinel(s SentinelType) *Node
Returns the list of column chunks in this row group. The chunks are
ordered in the order of leaf columns from the row group's schema.
If the underlying implementation is not read-only, the returned
parquet.ColumnChunk may implement other interfaces: for example,
parquet.ColumnBuffer if the chunk is backed by an in-memory buffer,
or typed writer interfaces like parquet.Int32Writer depending on the
underlying type of values that can be written to the chunk.
As an optimization, the row group may return the same slice across
multiple calls to this method. Applications should treat the returned
slice as read-only. DynamicColumns returns the concrete dynamic column names that were used
create its concrete parquet schema with a dynamic parquet schema. DynamicRows return an iterator over the rows in the row group. Returns the number of rows in the group.( ReleaseableRowGroup) Release() Returns a reader exposing the rows of the row group.
As an optimization, the returned parquet.Rows object may implement
parquet.RowWriterTo, and test the RowWriter it receives for an
implementation of the parquet.RowGroupWriter interface.
This optimization mechanism is leveraged by the parquet.CopyRows function
to skip the generic row-by-row copy algorithm and delegate the copy logic
to the parquet.Rows object. Returns the schema of rows in the group. Returns the list of sorting columns describing how rows are sorted in the
group.
The method will return an empty slice if the rows are not sorted.( ReleaseableRowGroup) String() string
ReleaseableRowGroup : github.com/polarsignals/frostdb/dynparquet.DynamicRowGroup
ReleaseableRowGroup : github.com/polarsignals/frostdb/query/expr.Particulate
ReleaseableRowGroup : github.com/parquet-go/parquet-go.RowGroup
ReleaseableRowGroup : expvar.Var
ReleaseableRowGroup : fmt.Stringer
NewList creates a new part list using atomic constructs.
NewLSM returns an LSM-like index of len(levels) levels.
wait is a function that will block until the given transaction has been committed; this is used only during compaction to ensure
that all the tx in the level up to the compaction tx have been committed before compacting.
The pages are generated with Goldsv0.8.2. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.
