package table

Import Path
	github.com/polarsignals/iceberg-go/table (on go.dev)

Dependency Relation
	imports 24 packages, and imported by 2 packages


Package-Level Type Names (total 20)
/* sort by: | */
type Identifier = ([])
Metadata for an iceberg table as specified in the Iceberg spec https://iceberg.apache.org/spec/#iceberg-table-spec CurrentSchema returns the table's current schema. CurrentSnapshot returns the table's current snapshot. DefaultPartitionSpec is the ID of the current spec that writers should use by default. GetMetadataLog is a list of metadata log entries. This is used to track GetSnapshotLog is a list of snapshot log entries. This is used to track LastColumnID returns the highest assigned column ID for the table. This is used to ensure fields are always assigned an unused ID when evolving schemas. LastPartitionSpecID is the highest assigned partition field ID across all partition specs for the table. This is used to ensure partition fields are always assigned an unused ID when evolving specs. LastUpdatedMillis is the timestamp in milliseconds from the unix epoch when the table was last updated. Each table metadata file should update this field just before writing. Location is the table's base location. This is used by writers to determine where to store data files, manifest files, and table metadata files. PartitionSpec returns the current partition spec that the table is using. PartitionSpecs returns the list of all partition specs in the table. Properties is a string to string map of table properties. This is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata. For example, commit.retry.num-retries is used to control the number of commit retries. SchemaID returns the ID of the current schema. Schemas returns the list of schemas, stored as objects with their schema-id. SnapshotByID find and return a specific snapshot by its ID. Returns nil if the ID is not found in the list of snapshots. SnapshotByName searches the list of snapshots for a snapshot with a given ref name. Returns nil if there's no ref with this name for a snapshot. SnapshotID returns the ID of the current snapshot. SnapshotRefs is a map of snapshot ref names to snapshot refs. This is used to Snapshots returns the list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. SortOrder returns the table's current sort order, ie: the one with the ID that matches the default-sort-order-id. SortOrderID returns the ID of the current sort order. SortOrders returns the list of sort orders in the table. TableUUID returns a UUID that identifies the table, generated when the table is created. Implementations must throw an exception if a table's UUID does not match the expected UUID after refreshing metadata. Version indicates the version of this metadata, 1 for V1, 2 for V2, etc. *MetadataV1 MetadataV1Builder *MetadataV2 func ParseMetadata(r io.Reader) (Metadata, error) func ParseMetadataBytes(b []byte) (Metadata, error) func ParseMetadataString(s string) (Metadata, error) func (*MetadataV1Builder).Build() Metadata func Table.Metadata() Metadata func CloneMetadataV1(m Metadata) *MetadataV1Builder func New(ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table func NewHDFSTable(ver int, ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table
MetadataFile string TimestampMs int64 func Metadata.GetMetadataLog() []MetadataLogEntry func (*MetadataV1Builder).WithMetadataLog(metadataLog []MetadataLogEntry) *MetadataV1Builder
commonMetadata.CurrentSchemaID int commonMetadata.CurrentSnapshotID *int64 commonMetadata.DefaultSortOrderID int commonMetadata.DefaultSpecID int commonMetadata.FormatVersion int commonMetadata.LastColumnId int commonMetadata.LastPartitionID *int commonMetadata.LastUpdatedMS int64 commonMetadata.Loc string commonMetadata.MetadataLog []MetadataLogEntry commonMetadata.Props iceberg.Properties commonMetadata.Refs map[string]SnapshotRef commonMetadata.SchemaList []*iceberg.Schema commonMetadata.SnapshotList []Snapshot commonMetadata.SnapshotLog []SnapshotLogEntry commonMetadata.SortOrderList []SortOrder commonMetadata.Specs []iceberg.PartitionSpec commonMetadata.UUID uuid.UUID Partition []iceberg.PartitionField Schema *iceberg.Schema (*MetadataV1) CurrentSchema() *iceberg.Schema (*MetadataV1) CurrentSnapshot() *Snapshot (*MetadataV1) DefaultPartitionSpec() int (*MetadataV1) GetMetadataLog() []MetadataLogEntry (*MetadataV1) GetSnapshotLog() []SnapshotLogEntry (*MetadataV1) LastColumnID() int (*MetadataV1) LastPartitionSpecID() *int (*MetadataV1) LastUpdatedMillis() int64 (*MetadataV1) Location() string (*MetadataV1) PartitionSpec() iceberg.PartitionSpec (*MetadataV1) PartitionSpecs() []iceberg.PartitionSpec (*MetadataV1) Properties() iceberg.Properties (*MetadataV1) SchemaID() int (*MetadataV1) Schemas() []*iceberg.Schema (*MetadataV1) SnapshotByID(id int64) *Snapshot (*MetadataV1) SnapshotByName(name string) *Snapshot (*MetadataV1) SnapshotID() *int64 (*MetadataV1) SnapshotRefs() map[string]SnapshotRef (*MetadataV1) Snapshots() []Snapshot (*MetadataV1) SortOrder() SortOrder (*MetadataV1) SortOrderID() int (*MetadataV1) SortOrders() []SortOrder (*MetadataV1) TableUUID() uuid.UUID (*MetadataV1) ToV2() MetadataV2 (*MetadataV1) UnmarshalJSON(b []byte) error (*MetadataV1) Version() int *MetadataV1 : Metadata *MetadataV1 : github.com/goccy/go-json.Unmarshaler *MetadataV1 : encoding/json.Unmarshaler
MetadataV1 *MetadataV1 MetadataV1.commonMetadata.CurrentSchemaID int MetadataV1.commonMetadata.CurrentSnapshotID *int64 MetadataV1.commonMetadata.DefaultSortOrderID int MetadataV1.commonMetadata.DefaultSpecID int MetadataV1.commonMetadata.FormatVersion int MetadataV1.commonMetadata.LastColumnId int MetadataV1.commonMetadata.LastPartitionID *int MetadataV1.commonMetadata.LastUpdatedMS int64 MetadataV1.commonMetadata.Loc string MetadataV1.commonMetadata.MetadataLog []MetadataLogEntry MetadataV1.commonMetadata.Props iceberg.Properties MetadataV1.commonMetadata.Refs map[string]SnapshotRef MetadataV1.commonMetadata.SchemaList []*iceberg.Schema MetadataV1.commonMetadata.SnapshotList []Snapshot MetadataV1.commonMetadata.SnapshotLog []SnapshotLogEntry MetadataV1.commonMetadata.SortOrderList []SortOrder MetadataV1.commonMetadata.Specs []iceberg.PartitionSpec MetadataV1.commonMetadata.UUID uuid.UUID MetadataV1.Partition []iceberg.PartitionField MetadataV1.Schema *iceberg.Schema (*MetadataV1Builder) Build() Metadata ( MetadataV1Builder) CurrentSchema() *iceberg.Schema ( MetadataV1Builder) CurrentSnapshot() *Snapshot ( MetadataV1Builder) DefaultPartitionSpec() int ( MetadataV1Builder) GetMetadataLog() []MetadataLogEntry ( MetadataV1Builder) GetSnapshotLog() []SnapshotLogEntry ( MetadataV1Builder) LastColumnID() int ( MetadataV1Builder) LastPartitionSpecID() *int ( MetadataV1Builder) LastUpdatedMillis() int64 ( MetadataV1Builder) Location() string ( MetadataV1Builder) PartitionSpec() iceberg.PartitionSpec ( MetadataV1Builder) PartitionSpecs() []iceberg.PartitionSpec ( MetadataV1Builder) Properties() iceberg.Properties ( MetadataV1Builder) SchemaID() int ( MetadataV1Builder) Schemas() []*iceberg.Schema ( MetadataV1Builder) SnapshotByID(id int64) *Snapshot ( MetadataV1Builder) SnapshotByName(name string) *Snapshot ( MetadataV1Builder) SnapshotID() *int64 ( MetadataV1Builder) SnapshotRefs() map[string]SnapshotRef ( MetadataV1Builder) Snapshots() []Snapshot ( MetadataV1Builder) SortOrder() SortOrder ( MetadataV1Builder) SortOrderID() int ( MetadataV1Builder) SortOrders() []SortOrder ( MetadataV1Builder) TableUUID() uuid.UUID ( MetadataV1Builder) ToV2() MetadataV2 ( MetadataV1Builder) UnmarshalJSON(b []byte) error ( MetadataV1Builder) Version() int WithCurrentSchemaID sets the optional current-schema-id field of the metadata. WithCurrentSnapshotID sets the optional current-snapshot-id field of the metadata. WithDefaultSortOrderID sets the optional default-sort-order-id field of the metadata. WithDefaultSpecID sets the optional default-spec-id field of the metadata. (*MetadataV1Builder) WithFormatVersion(version int) *MetadataV1Builder WithLastPartitionID sets the optional last-partition-id field of the metadata. (*MetadataV1Builder) WithLastUpdatedMs(lastUpdatedMs int64) *MetadataV1Builder (*MetadataV1Builder) WithLocation(location string) *MetadataV1Builder WithMetadataLog sets the optional metadata-log field of the metadata. WithProperties sets the optional partition-specs field of the metadata. WithProperties sets the optional properties field of the metadata. (*MetadataV1Builder) WithSchema(schema *iceberg.Schema) *MetadataV1Builder WithSchemas sets the optional schemas field of the metadata. WithSnapshotLog sets the optional snapshot-log field of the metadata. WithSnapshots sets the optional snapshots field of the metadata. WithSortOrders sets the optional sort-orders field of the metadata. WithTableUUID sets the optional table-uuid field of the metadata. MetadataV1Builder : Metadata MetadataV1Builder : github.com/goccy/go-json.Unmarshaler MetadataV1Builder : encoding/json.Unmarshaler func CloneMetadataV1(m Metadata) *MetadataV1Builder func NewMetadataV1Builder(location string, schema *iceberg.Schema, lastUpdatesMs int64, lastColumnId int) *MetadataV1Builder func (*MetadataV1Builder).WithCurrentSchemaID(currentSchemaID int) *MetadataV1Builder func (*MetadataV1Builder).WithCurrentSnapshotID(currentSnapshotID int64) *MetadataV1Builder func (*MetadataV1Builder).WithDefaultSortOrderID(defaultSortOrderID int) *MetadataV1Builder func (*MetadataV1Builder).WithDefaultSpecID(defaultSpecID int) *MetadataV1Builder func (*MetadataV1Builder).WithFormatVersion(version int) *MetadataV1Builder func (*MetadataV1Builder).WithLastPartitionID(lastPartitionID int) *MetadataV1Builder func (*MetadataV1Builder).WithLastUpdatedMs(lastUpdatedMs int64) *MetadataV1Builder func (*MetadataV1Builder).WithLocation(location string) *MetadataV1Builder func (*MetadataV1Builder).WithMetadataLog(metadataLog []MetadataLogEntry) *MetadataV1Builder func (*MetadataV1Builder).WithPartitionSpecs(specs []iceberg.PartitionSpec) *MetadataV1Builder func (*MetadataV1Builder).WithProperties(properties iceberg.Properties) *MetadataV1Builder func (*MetadataV1Builder).WithSchema(schema *iceberg.Schema) *MetadataV1Builder func (*MetadataV1Builder).WithSchemas(schemas []*iceberg.Schema) *MetadataV1Builder func (*MetadataV1Builder).WithSnapshotLog(snapshotLog []SnapshotLogEntry) *MetadataV1Builder func (*MetadataV1Builder).WithSnapshots(snapshots []Snapshot) *MetadataV1Builder func (*MetadataV1Builder).WithSortOrders(sortOrders []SortOrder) *MetadataV1Builder func (*MetadataV1Builder).WithTableUUID(id uuid.UUID) *MetadataV1Builder
commonMetadata.CurrentSchemaID int commonMetadata.CurrentSnapshotID *int64 commonMetadata.DefaultSortOrderID int commonMetadata.DefaultSpecID int commonMetadata.FormatVersion int commonMetadata.LastColumnId int commonMetadata.LastPartitionID *int commonMetadata.LastUpdatedMS int64 commonMetadata.Loc string commonMetadata.MetadataLog []MetadataLogEntry commonMetadata.Props iceberg.Properties commonMetadata.Refs map[string]SnapshotRef commonMetadata.SchemaList []*iceberg.Schema commonMetadata.SnapshotList []Snapshot commonMetadata.SnapshotLog []SnapshotLogEntry commonMetadata.SortOrderList []SortOrder commonMetadata.Specs []iceberg.PartitionSpec commonMetadata.UUID uuid.UUID LastSequenceNumber int (*MetadataV2) CurrentSchema() *iceberg.Schema (*MetadataV2) CurrentSnapshot() *Snapshot (*MetadataV2) DefaultPartitionSpec() int (*MetadataV2) GetMetadataLog() []MetadataLogEntry (*MetadataV2) GetSnapshotLog() []SnapshotLogEntry (*MetadataV2) LastColumnID() int (*MetadataV2) LastPartitionSpecID() *int (*MetadataV2) LastUpdatedMillis() int64 (*MetadataV2) Location() string (*MetadataV2) PartitionSpec() iceberg.PartitionSpec (*MetadataV2) PartitionSpecs() []iceberg.PartitionSpec (*MetadataV2) Properties() iceberg.Properties (*MetadataV2) SchemaID() int (*MetadataV2) Schemas() []*iceberg.Schema (*MetadataV2) SnapshotByID(id int64) *Snapshot (*MetadataV2) SnapshotByName(name string) *Snapshot (*MetadataV2) SnapshotID() *int64 (*MetadataV2) SnapshotRefs() map[string]SnapshotRef (*MetadataV2) Snapshots() []Snapshot (*MetadataV2) SortOrder() SortOrder (*MetadataV2) SortOrderID() int (*MetadataV2) SortOrders() []SortOrder (*MetadataV2) TableUUID() uuid.UUID (*MetadataV2) UnmarshalJSON(b []byte) error (*MetadataV2) Version() int *MetadataV2 : Metadata *MetadataV2 : github.com/goccy/go-json.Unmarshaler *MetadataV2 : encoding/json.Unmarshaler func (*MetadataV1).ToV2() MetadataV2
const NullsFirst const NullsLast
func ValidOperation(s string) (Operation, error) const OpAppend const OpDelete const OpOverwrite const OpReplace
( ReadOnlyTable) Bucket() objstore.Bucket ( ReadOnlyTable) CurrentSnapshot() *Snapshot ( ReadOnlyTable) Equals(other Table) bool ( ReadOnlyTable) Identifier() Identifier ( ReadOnlyTable) Location() string ( ReadOnlyTable) Metadata() Metadata ( ReadOnlyTable) MetadataLocation() string ( ReadOnlyTable) Properties() iceberg.Properties ( ReadOnlyTable) Schema() *iceberg.Schema ( ReadOnlyTable) Schemas() map[int]*iceberg.Schema ( ReadOnlyTable) SnapshotByID(id int64) *Snapshot ( ReadOnlyTable) SnapshotByName(name string) *Snapshot (*ReadOnlyTable) SnapshotWriter(options ...WriterOption) (SnapshotWriter, error) ( ReadOnlyTable) SortOrder() SortOrder ( ReadOnlyTable) Spec() iceberg.PartitionSpec *ReadOnlyTable : Table
RefType will be either a BranchRef or a TagRef const BranchRef const TagRef
ManifestList string ParentSnapshotID *int64 SchemaID *int SequenceNumber int64 SnapshotID int64 Summary *Summary TimestampMs int64 ( Snapshot) Equals(other Snapshot) bool ( Snapshot) Manifests(bucket objstore.Bucket) ([]iceberg.ManifestFile, error) ( Snapshot) String() string Snapshot : expvar.Var Snapshot : fmt.Stringer func Metadata.CurrentSnapshot() *Snapshot func Metadata.SnapshotByID(int64) *Snapshot func Metadata.SnapshotByName(name string) *Snapshot func Metadata.Snapshots() []Snapshot func Table.CurrentSnapshot() *Snapshot func Table.SnapshotByID(id int64) *Snapshot func Table.SnapshotByName(name string) *Snapshot func (*MetadataV1Builder).WithSnapshots(snapshots []Snapshot) *MetadataV1Builder func Snapshot.Equals(other Snapshot) bool
SnapshotID int64 TimestampMs int64 func Metadata.GetSnapshotLog() []SnapshotLogEntry func (*MetadataV1Builder).WithSnapshotLog(snapshotLog []SnapshotLogEntry) *MetadataV1Builder
SnapshotRef represents the reference information for a specific snapshot MaxRefAgeMs *int64 MaxSnapshotAgeMs *int64 MinSnapshotsToKeep *int SnapshotID int64 SnapshotRefType RefType (*SnapshotRef) UnmarshalJSON(b []byte) error *SnapshotRef : github.com/goccy/go-json.Unmarshaler *SnapshotRef : encoding/json.Unmarshaler func Metadata.SnapshotRefs() map[string]SnapshotRef
SnapshotWriter is an interface for writing a new snapshot to a table. Append accepts a ReaderAt object that should read the Parquet file that is to be added to the snapshot. Close writes the new snapshot to the table and closes the writer. It is an error to call Append after Close. DeleteDataFile deletes a data file from the table. The function passed to this method should return true if the file should be deleted. SnapshotWriter : github.com/tetratelabs/wazero.CompilationCache SnapshotWriter : github.com/tetratelabs/wazero/api.Closer func (*ReadOnlyTable).SnapshotWriter(options ...WriterOption) (SnapshotWriter, error) func Table.SnapshotWriter(options ...WriterOption) (SnapshotWriter, error)
const SortASC const SortDESC
SortField describes a field used in a sort order definition. Direction is an enum indicating ascending or descending direction. NullOrder describes the order of null values when sorting should be only either nulls-first or nulls-last enum values. SourceID is the source column id from the table's schema Transform is the tranformation used to produce values to be sorted on from the source column. (*SortField) MarshalJSON() ([]byte, error) (*SortField) String() string (*SortField) UnmarshalJSON(b []byte) error *SortField : github.com/goccy/go-json.Marshaler *SortField : github.com/goccy/go-json.Unmarshaler *SortField : encoding/json.Marshaler *SortField : encoding/json.Unmarshaler *SortField : expvar.Var *SortField : fmt.Stringer
SortOrder describes how the data is sorted within the table. Data can be sorted within partitions by columns to gain performance. The order of the sort fields within the list defines the order in which the sort is applied to the data. Fields []SortField OrderID int ( SortOrder) String() string (*SortOrder) UnmarshalJSON(b []byte) error *SortOrder : github.com/goccy/go-json.Unmarshaler *SortOrder : encoding/json.Unmarshaler SortOrder : expvar.Var SortOrder : fmt.Stringer func Metadata.SortOrder() SortOrder func Metadata.SortOrders() []SortOrder func Table.SortOrder() SortOrder func (*MetadataV1Builder).WithSortOrders(sortOrders []SortOrder) *MetadataV1Builder var UnsortedSortOrder
Summary stores the summary information for a snapshot indicating the operation that created the snapshot, and various properties which might exist in the summary. Operation Operation Properties map[string]string (*Summary) Equals(other *Summary) bool (*Summary) MarshalJSON() ([]byte, error) (*Summary) String() string (*Summary) UnmarshalJSON(b []byte) (err error) *Summary : github.com/goccy/go-json.Marshaler *Summary : github.com/goccy/go-json.Unmarshaler *Summary : encoding/json.Marshaler *Summary : encoding/json.Unmarshaler *Summary : expvar.Var *Summary : fmt.Stringer func (*Summary).Equals(other *Summary) bool
( Table) Bucket() objstore.Bucket ( Table) CurrentSnapshot() *Snapshot ( Table) Equals(other Table) bool ( Table) Identifier() Identifier ( Table) Location() string ( Table) Metadata() Metadata ( Table) MetadataLocation() string ( Table) Properties() iceberg.Properties ( Table) Schema() *iceberg.Schema ( Table) Schemas() map[int]*iceberg.Schema ( Table) SnapshotByID(id int64) *Snapshot ( Table) SnapshotByName(name string) *Snapshot ( Table) SnapshotWriter(options ...WriterOption) (SnapshotWriter, error) ( Table) SortOrder() SortOrder ( Table) Spec() iceberg.PartitionSpec *ReadOnlyTable func New(ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table func NewFromLocation(ident Identifier, metalocation string, bucket objstore.Bucket) (Table, error) func NewHDFSTable(ver int, ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table func github.com/polarsignals/iceberg-go/catalog.Catalog.CreateTable(ctx context.Context, location string, schema *iceberg.Schema, props iceberg.Properties, options ...catalog.TableOption) (Table, error) func github.com/polarsignals/iceberg-go/catalog.Catalog.LoadTable(ctx context.Context, identifier Identifier, props iceberg.Properties) (Table, error) func github.com/polarsignals/iceberg-go/catalog.Catalog.RenameTable(ctx context.Context, from, to Identifier) (Table, error) func DeleteOrphanFiles(ctx context.Context, table Table, age time.Duration) error func NewSnapshotWriter(commit func(ctx context.Context, v int) error, version int, bucket objstore.Bucket, table Table, options ...WriterOption) snapshotWriter func Table.Equals(other Table) bool
func WithExpireSnapshotsOlderThan(d time.Duration) WriterOption func WithFastAppend() WriterOption func WithLogger(logger log.Logger) WriterOption func WithManifestSizeBytes(size int) WriterOption func WithMergeSchema() WriterOption func WithMetadataDeleteAfterCommit() WriterOption func WithMetadataPreviousVersionsMax(n int) WriterOption func NewSnapshotWriter(commit func(ctx context.Context, v int) error, version int, bucket objstore.Bucket, table Table, options ...WriterOption) snapshotWriter func (*ReadOnlyTable).SnapshotWriter(options ...WriterOption) (SnapshotWriter, error) func Table.SnapshotWriter(options ...WriterOption) (SnapshotWriter, error)
Package-Level Functions (total 18)
DeleteOrphanFiles will delete all files in the table's bucket that are not pointed to by any snapshot in the table. NOTE: https://iceberg.apache.org/docs/latest/maintenance/#delete-orphan-files "It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete because it might corrupt the table if in-progress files are considered orphaned and are deleted. The default interval is 3 days."
func New(ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table
func NewFromLocation(ident Identifier, metalocation string, bucket objstore.Bucket) (Table, error)
func NewHDFSTable(ver int, ident Identifier, meta Metadata, location string, bucket objstore.Bucket) Table
NewMetadataV1Builder returns a new MetadataV1Builder.
func NewSnapshotWriter(commit func(ctx context.Context, v int) error, version int, bucket objstore.Bucket, table Table, options ...WriterOption) snapshotWriter
ParseMetadata parses json metadata provided by the passed in reader, returning an error if one is encountered.
ParseMetadataBytes is like [ParseMetadataString] but for a byte slice.
ParseMetadataString is like [ParseMetadata], but for a string rather than an io.Reader.
ValidOperation ensures that a given string is one of the valid operation types: append,replace,overwrite,delete
Package-Level Constants (total 13)
const BranchRef RefType = "branch"
const MainBranch = "main"
const NullsFirst NullOrder = "nulls-first"
const NullsLast NullOrder = "nulls-last"
const OpAppend Operation = "append"
const OpDelete Operation = "delete"
const OpOverwrite Operation = "overwrite"
const OpReplace Operation = "replace"
const SortASC SortDirection = "asc"
const SortDESC SortDirection = "desc"
const TagRef RefType = "tag"