Close persists all data from the columnstore to storage.
It is no longer valid to use the coumnstore for reads or writes, and the object should not longer be reused. DB gets or creates a database on the given ColumnStore with the given
options. Note that if the database already exists, the options will be
applied cumulatively to the database. DBs returns all the DB names of this column store.(*ColumnStore) DatabasesDir() string(*ColumnStore) DropDB(name string) error(*ColumnStore) GetDB(name string) (*DB, error)
*ColumnStore : github.com/prometheus/common/expfmt.Closer
*ColumnStore : io.Closer
func New(options ...Option) (*ColumnStore, error)
DefaultObjstoreBucket is the default implementation of the DataSource and DataSink interface.Bucketstorage.Bucket Attributes returns information about the specified object.( DefaultObjstoreBucket) Close() error Delete removes the object with the given name.
If object does not exist in the moment of deletion, Delete should throw error. Exists checks if the given object exists in the bucket. Get returns a reader for the given object name. GetRange returns a new range reader for the given object name and range.( DefaultObjstoreBucket) GetReaderAt(ctx context.Context, name string) (io.ReaderAt, error) IsAccessDeniedErr returns true if access to object is denied. IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations. Iter calls f for each entry in the given directory (not recursive.). The argument to f is the full
object name including the prefix of the inspected directory.
Entries are passed to function in sorted order. Name returns the bucket name for the provider.(*DefaultObjstoreBucket) Prefixes(ctx context.Context, prefix string) ([]string, error) ProcessFile will process a bucket block parquet file.(*DefaultObjstoreBucket) Scan(ctx context.Context, prefix string, _ *dynparquet.Schema, filter logicalplan.Expr, lastBlockTimestamp uint64, callback func(context.Context, any) error) error(*DefaultObjstoreBucket) String() string Upload the contents of the reader as an object into the bucket.
Upload should be idempotent.
*DefaultObjstoreBucket : DataSink
*DefaultObjstoreBucket : DataSinkSource
*DefaultObjstoreBucket : DataSource
DefaultObjstoreBucket : github.com/polarsignals/frostdb/query/logicalplan.Named
DefaultObjstoreBucket : github.com/polarsignals/frostdb/storage.Bucket
DefaultObjstoreBucket : github.com/prometheus/common/expfmt.Closer
DefaultObjstoreBucket : github.com/thanos-io/objstore.Bucket
DefaultObjstoreBucket : github.com/thanos-io/objstore.BucketReader
*DefaultObjstoreBucket : expvar.Var
*DefaultObjstoreBucket : fmt.Stringer
DefaultObjstoreBucket : io.Closer
func NewDefaultBucket(b storage.Bucket, options ...DefaultObjstoreBucketOption) *DefaultObjstoreBucket
func NewDefaultObjstoreBucket(b objstore.Bucket, options ...DefaultObjstoreBucketOption) *DefaultObjstoreBucket
Type Parameters:
T: any GenericTable is a wrapper around *Table that writes structs of type T. It
consist of a generic arrow.Record builder that ingests structs of type T.
The generated record is then passed to (*Table).InsertRecord.
Struct tag `frostdb` is used to pass options for the schema for T.
This api is opinionated.
- Nested Columns are not supported
# Tags
Use `frostdb` to define tags that customizes field values. You can express
everything needed to construct schema v1alpha1.
Tags are defined as a comma separated list. The first item is the column
name. Column name is optional, when omitted it is derived from the field name
(snake_cased)
Supported Tags
delta_binary_packed | Delta binary packed encoding.
brotli | Brotli compression.
asc | Sorts in ascending order.Use asc(n) where n is an integer for sorting order.
gzip | GZIP compression.
snappy | Snappy compression.
delta_length_byte_array | Delta Length Byte Array encoding.
delta_byte_array | Delta Byte Array encoding.
desc | Sorts in descending order.Use desc(n) where n is an integer for sorting order
lz4_raw | LZ4_RAW compression.
pre_hash | Prehash the column before storing it.
null_first | When used wit asc nulls are smallest and with des nulls are largest.
zstd | ZSTD compression.
rle_dict | Dictionary run-length encoding.
plain | Plain encoding.
Example tagged Sample struct
type Sample struct {
ExampleType string `frostdb:"example_type,rle_dict,asc(0)"`
Labels []Label `frostdb:"labels,rle_dict,null,dyn,asc(1),null_first"`
Stacktrace []uuid.UUID `frostdb:"stacktrace,rle_dict,asc(3),null_first"`
Timestamp int64 `frostdb:"timestamp,asc(2)"`
Value int64 `frostdb:"value"`
}
# Dynamic columns
Field of type map<string, T> is a dynamic column by default.
type Example struct {
// Use supported tags to customize the column value
Labels map[string]string `frostdb:"labels"`
}
# Repeated columns
Fields of type []int64, []float64, []bool, and []string are supported. These
are represented as arrow.LIST.
Generated schema for the repeated columns applies all supported tags. By
default repeated fields are nullable. You can safely pass nil slices for
repeated columns.Table*Table( GenericTable) ActiveBlock() *TableBlock( GenericTable) ActiveWriteBlock() (*TableBlock, func(), error)( GenericTable) EnsureCompaction() error IndexConfig returns the index configuration for the table. It makes a copy of the column store index config and injects it's compactParts method.( GenericTable) InsertRecord(ctx context.Context, record arrow.Record) (uint64, error) Iterator iterates in order over all granules in the table. It stops iterating when the iterator function returns false.(*GenericTable[T]) Release()( GenericTable) RotateBlock(_ context.Context, block *TableBlock, opts ...RotateBlockOption) error( GenericTable) Schema() *dynparquet.Schema SchemaIterator iterates in order over all granules in the table and returns
all the schemas seen across the table.( GenericTable) View(ctx context.Context, fn func(ctx context.Context, tx uint64) error) error Write builds arrow.Record directly from values and calls (*Table).InsertRecord.
GenericTable : github.com/polarsignals/frostdb/query/logicalplan.TableReader
func NewGenericTable[T](db *DB, name string, mem memory.Allocator, options ...TableOption) (*GenericTable[T], error)
EnsureCompaction forces a TableBlock compaction. Index provides atomic access to the table index.(*TableBlock) InsertRecord(_ context.Context, tx uint64, record arrow.Record) error Persist uploads the block to the underlying bucket. Serialize the table block into a single Parquet file. Size returns the cumulative size of all buffers in the table. This is roughly the size of the table in bytes.
func (*Table).ActiveBlock() *TableBlock
func (*Table).ActiveWriteBlock() (*TableBlock, func(), error)
func (*Table).RotateBlock(_ context.Context, block *TableBlock, opts ...RotateBlockOption) error
Insert performs an insertion sort of the given tx.(*TxPool) Iterate(iterate func(txn uint64) bool) Stop stops the TxPool's cleaner goroutine.
func NewTxPool(watermark *atomic.Uint64) *TxPool
( WAL) Close() error( WAL) FirstIndex() (uint64, error)( WAL) LastIndex() (uint64, error)( WAL) Log(tx uint64, record *walpb.Record) error( WAL) LogRecord(tx uint64, table string, record arrow.Record) error Replay replays WAL records from the given first index. If firstIndex is
0, the first index read from the WAL is used (i.e. given a truncation,
using 0 is still valid). If the given firstIndex is less than the WAL's
first index on disk, the replay happens from the first index on disk.
If the handler panics, the WAL implementation will truncate the WAL up to
the last valid index.( WAL) Reset(nextTx uint64) error( WAL) Truncate(tx uint64) error
*github.com/polarsignals/frostdb/wal.FileWAL
*github.com/polarsignals/frostdb/wal.NopWAL
WAL : github.com/prometheus/common/expfmt.Closer
WAL : io.Closer
Package-Level Functions (total 43)
DefaultIndexConfig returns the default level configs used. This is a function
So that any modifications to the result will not affect the default config.
FromConfig sets the table configuration from the given config.
NOTE: that this does not override the schema even though that is included in the passed in config.
NewTxPool returns a new TxPool and starts the pool cleaner routine.
The transaction pool is used to keep track of completed transactions. It does
this by inserting completed transactions into an ordered linked list.
Ex:
insert: 12
[9]->[10]->[13] => [9]->[10]->[12]->[13]
Inserting a new node triggers the pool cleaner routine to run. The pool
cleaner's job is to increment a high-watermark counter when it encounters
contiguous transactions in the list, and then remove those elements in the
pool.
Ex:
watermark: 7 insert: 8
[9]->[10]->[13] => [8]->[9]->[10]->[13] (cleaner notified)
[8]->[9]->[10]->[13]
^ watermark++; delete 8
[9]->[10]->[13]
^ watermark++; delete 9
[10]->[13]
^ watermark++; delete 9
[13]
watermark: 10
TxPool is a sorted lockless linked-list described in
https://timharris.uk/papers/2001-disc.pdf
WithRecoveryConcurrency limits the number of databases that are recovered
simultaneously when calling frostdb.New. This helps limit memory usage on
recovery.
WithRotateBlockSkipPersist instructs the block rotation operation to not
persist the block to object storage.
WithRotateBlockWaitGroup provides a WaitGroup. The rotate block operation
will call wg.Done once the block has been persisted. Otherwise, RotateBlock
asynchronously persists the block.
WithRowGroupSize sets the size in number of rows for each row group for parquet files. A <= 0 value indicates no limit.
WithSnapshotTriggerSize specifies a size in bytes of uncompressed inserts
that will trigger a snapshot of the whole database. This can be larger than
the active memory size given that the active memory size tracks the size of
*compressed* data, while snapshots are triggered based on the *uncompressed*
data inserted into the database. The reason this choice was made is that
if a database instance crashes, it is forced to reread all uncompressed
inserts since the last snapshot from the WAL, which could potentially lead
to unrecoverable OOMs on startup. Defining the snapshot trigger in terms of
uncompressed bytes limits the memory usage on recovery to at most the
snapshot trigger size (as long as snapshots were successful).
If 0, snapshots are disabled. Note that snapshots (if enabled) are also
triggered on block rotation of any database table.
Snapshots are complementary to the WAL and will also be disabled if the WAL
is disabled.
The pages are generated with Goldsv0.8.2. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.
