package schema

Import Path
	github.com/apache/arrow-go/v18/parquet/schema (on go.dev)

Dependency Relation
	imports 16 packages, and imported by one package

Involved Source Files column.go converted_types.go helpers.go logical_types.go node.go reflection.go Package schema provides types and functions for manipulating and building parquet file schemas. Some of the utilities provided include building a schema using Struct Tags on a struct type, getting Column Paths from a node, and dealing with the converted and logical types for Parquet. Logical types specify ways to interpret the primitive types allowing the number of primitive types to be smaller and reuse efficient encodings. For instance a "string" is just a ByteArray column with a UTF-8 annotation or "String Logical Type". For more information about Logical and Converted Types, check: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
Package-Level Type Names (total 33)
/* sort by: | */
BSONLogicalType represents a binary JSON string in the byte array ( BSONLogicalType) Equals(rhs LogicalType) bool ( BSONLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( BSONLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( BSONLogicalType) IsNested() bool ( BSONLogicalType) IsNone() bool ( BSONLogicalType) IsSerialized() bool ( BSONLogicalType) IsValid() bool ( BSONLogicalType) MarshalJSON() ([]byte, error) ( BSONLogicalType) SortOrder() SortOrder ( BSONLogicalType) String() string ( BSONLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) BSONLogicalType : LogicalType BSONLogicalType : github.com/goccy/go-json.Marshaler BSONLogicalType : database/sql/driver.Validator BSONLogicalType : encoding/json.Marshaler BSONLogicalType : expvar.Var BSONLogicalType : fmt.Stringer
Column encapsulates the information necessary to interpret primitive column data in the context of a particular schema. We have to examine the node structure of a column's path to the root in the schema tree to be able to reassemble the nested structure from the repetition and definition levels. (*Column) ColumnOrder() parquet.ColumnOrder ColumnPath returns the full path to this column from the root of the schema (*Column) ConvertedType() ConvertedType Equals will return true if the rhs Column has the same Max Repetition and Definition levels along with having the same node definition. (*Column) LogicalType() LogicalType (*Column) MaxDefinitionLevel() int16 (*Column) MaxRepetitionLevel() int16 Name is the column's name Path is equivalent to ColumnPath().String() returning the dot-string version of the path (*Column) PhysicalType() parquet.Type SchemaNode returns the underlying Node in the schema tree for this column. SortOrder returns the sort order of this column's statistics based on the Logical and Converted types. (*Column) String() string TypeLength is -1 if not a FixedLenByteArray, otherwise it is the length of elements in the column *Column : github.com/polarsignals/frostdb/query/logicalplan.Named *Column : expvar.Var *Column : fmt.Stringer func NewColumn(n *PrimitiveNode, maxDefinitionLvl, maxRepetitionLvl int16) *Column func (*Schema).Column(i int) *Column func (*Column).Equals(rhs *Column) bool
ConvertedType corresponds to the ConvertedType in the parquet.Thrift, with added values of None and NA for handling when these values are not set in the metadata ( ConvertedType) String() string ToLogicalType returns the correct LogicalType for the given ConvertedType, using the decimal metadata provided to define the precision/scale if necessary ConvertedType : expvar.Var ConvertedType : fmt.Stringer func BSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func (*Column).ConvertedType() ConvertedType func DateLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func DecimalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func EnumLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func Float16LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func IntervalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func IntLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func JSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func ListLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func MapLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func Node.ConvertedType() ConvertedType func NoLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func NullLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func StringLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TemporalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TimeLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TimestampLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func UnknownLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func UUIDLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func VariantLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func GetSortOrder(convert ConvertedType, primitive format.Type) SortOrder func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error) func NewPrimitiveNodeConverted(name string, repetition parquet.Repetition, typ parquet.Type, converted ConvertedType, typeLen, precision, scale int, id int32) (*PrimitiveNode, error) func BSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func DateLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func DecimalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func EnumLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func Float16LogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func IntervalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func IntLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func JSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func ListLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func LogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool func MapLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func NoLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func NullLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func StringLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func TemporalLogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool func TimeLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func TimestampLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func UnknownLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func UUIDLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func VariantLogicalType.IsCompatible(ct ConvertedType, _ DecimalMetadata) bool
DateLogicalType is an int32 representing the number of days since the Unix Epoch 1 January 1970 ( DateLogicalType) Equals(rhs LogicalType) bool ( DateLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( DateLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool ( DateLogicalType) IsNested() bool ( DateLogicalType) IsNone() bool ( DateLogicalType) IsSerialized() bool ( DateLogicalType) IsValid() bool ( DateLogicalType) MarshalJSON() ([]byte, error) ( DateLogicalType) SortOrder() SortOrder ( DateLogicalType) String() string ( DateLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) DateLogicalType : LogicalType DateLogicalType : github.com/goccy/go-json.Marshaler DateLogicalType : database/sql/driver.Validator DateLogicalType : encoding/json.Marshaler DateLogicalType : expvar.Var DateLogicalType : fmt.Stringer
DecimalLogicalType is used to represent a decimal value of a given precision and scale ( DecimalLogicalType) Equals(rhs LogicalType) bool ( DecimalLogicalType) IsApplicable(typ parquet.Type, tlen int32) bool ( DecimalLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( DecimalLogicalType) IsNested() bool ( DecimalLogicalType) IsNone() bool ( DecimalLogicalType) IsSerialized() bool ( DecimalLogicalType) IsValid() bool ( DecimalLogicalType) MarshalJSON() ([]byte, error) ( DecimalLogicalType) Precision() int32 ( DecimalLogicalType) Scale() int32 ( DecimalLogicalType) SortOrder() SortOrder ( DecimalLogicalType) String() string ( DecimalLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) DecimalLogicalType : LogicalType DecimalLogicalType : github.com/goccy/go-json.Marshaler DecimalLogicalType : database/sql/driver.Validator DecimalLogicalType : encoding/json.Marshaler DecimalLogicalType : expvar.Var DecimalLogicalType : fmt.Stringer
DecimalMetadata is a struct for managing scale and precision information between converted and logical types. IsSet bool Precision int32 Scale int32 func BSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func DateLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func DecimalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func EnumLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func Float16LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func IntervalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func IntLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func JSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func ListLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func MapLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func NoLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func NullLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func (*PrimitiveNode).DecimalMetadata() DecimalMetadata func StringLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TemporalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TimeLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func TimestampLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func UnknownLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func UUIDLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func VariantLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata) func BSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func ConvertedType.ToLogicalType(convertedDecimal DecimalMetadata) LogicalType func DateLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func DecimalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func EnumLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func Float16LogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func IntervalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func IntLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func JSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func ListLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func LogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool func MapLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func NoLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func NullLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func StringLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool func TemporalLogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool func TimeLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func TimestampLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func UnknownLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func UUIDLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool func VariantLogicalType.IsCompatible(ct ConvertedType, _ DecimalMetadata) bool
EnumLogicalType is for representing an enum, which should be a byte array type ( EnumLogicalType) Equals(rhs LogicalType) bool ( EnumLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( EnumLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool ( EnumLogicalType) IsNested() bool ( EnumLogicalType) IsNone() bool ( EnumLogicalType) IsSerialized() bool ( EnumLogicalType) IsValid() bool ( EnumLogicalType) MarshalJSON() ([]byte, error) ( EnumLogicalType) SortOrder() SortOrder ( EnumLogicalType) String() string ( EnumLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) EnumLogicalType : LogicalType EnumLogicalType : github.com/goccy/go-json.Marshaler EnumLogicalType : database/sql/driver.Validator EnumLogicalType : encoding/json.Marshaler EnumLogicalType : expvar.Var EnumLogicalType : fmt.Stringer
FieldList is an alias for a slice of Nodes Len is equivalent to len(fieldlist) func GroupNodeFromThrift(elem *format.SchemaElement, fields FieldList) (*GroupNode, error) func NewGroupNode(name string, repetition parquet.Repetition, fields FieldList, fieldID int32) (*GroupNode, error) func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error) func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error)
Float16LogicalType can only be used with a FixedLength byte array column that is exactly 2 bytes long ( Float16LogicalType) Equals(rhs LogicalType) bool ( Float16LogicalType) IsApplicable(t parquet.Type, tlen int32) bool ( Float16LogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( Float16LogicalType) IsNested() bool ( Float16LogicalType) IsNone() bool ( Float16LogicalType) IsSerialized() bool ( Float16LogicalType) IsValid() bool ( Float16LogicalType) MarshalJSON() ([]byte, error) ( Float16LogicalType) SortOrder() SortOrder ( Float16LogicalType) String() string ( Float16LogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) Float16LogicalType : LogicalType Float16LogicalType : github.com/goccy/go-json.Marshaler Float16LogicalType : database/sql/driver.Validator Float16LogicalType : encoding/json.Marshaler Float16LogicalType : expvar.Var Float16LogicalType : fmt.Stringer
GroupNode is for managing nested nodes like List, Map, etc. (*GroupNode) ConvertedType() ConvertedType Equals will compare this node to the provided node and only return true if this node and all of it's children are the same as the passed in node and its children. Field returns the node in the field list which is of the provided (0-based) index (*GroupNode) FieldID() int32 FieldIndexByField looks up the index child of this node. Returns -1 if n isn't a child of this group FieldIndexByName provides the index for the field of the given name. Returns -1 if not found. If there are more than one field of this name, it returns the index for the first one. HasRepeatedFields returns true if any of the children of this node have Repeated as its repetition type. This is recursive and will check the children of any group nodes that are children. (*GroupNode) LogicalType() LogicalType (*GroupNode) Name() string NumFields returns the number of direct child fields for this group node (*GroupNode) Parent() Node (*GroupNode) Path() string (*GroupNode) RepetitionType() parquet.Repetition (*GroupNode) SetParent(p Node) (*GroupNode) Type() NodeType Visit is for implementing a Visitor pattern handler to walk a schema's tree. One example is the Schema Printer which walks the tree to print out the schema in order. *GroupNode : Node *GroupNode : github.com/polarsignals/frostdb/query/logicalplan.Named func GroupNodeFromThrift(elem *format.SchemaElement, fields FieldList) (*GroupNode, error) func ListOf(n Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error) func ListOfWithName(listName string, element Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error) func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error) func MustGroup(n Node, err error) *GroupNode func NewGroupNode(name string, repetition parquet.Repetition, fields FieldList, fieldID int32) (*GroupNode, error) func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error) func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error) func (*Schema).Root() *GroupNode func NewSchema(root *GroupNode) *Schema func ToThrift(schema *GroupNode) []*format.SchemaElement
IntervalLogicalType is not yet in the thrift spec, but represents an interval time and needs to be a fixed length byte array of 12 bytes ( IntervalLogicalType) Equals(rhs LogicalType) bool ( IntervalLogicalType) IsApplicable(t parquet.Type, tlen int32) bool ( IntervalLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( IntervalLogicalType) IsNested() bool ( IntervalLogicalType) IsNone() bool ( IntervalLogicalType) IsSerialized() bool ( IntervalLogicalType) IsValid() bool ( IntervalLogicalType) MarshalJSON() ([]byte, error) ( IntervalLogicalType) SortOrder() SortOrder ( IntervalLogicalType) String() string ( IntervalLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) IntervalLogicalType : LogicalType IntervalLogicalType : github.com/goccy/go-json.Marshaler IntervalLogicalType : database/sql/driver.Validator IntervalLogicalType : encoding/json.Marshaler IntervalLogicalType : expvar.Var IntervalLogicalType : fmt.Stringer
IntLogicalType represents an integer type of a specific bit width and is either signed or unsigned. ( IntLogicalType) BitWidth() int8 ( IntLogicalType) Equals(rhs LogicalType) bool ( IntLogicalType) IsApplicable(typ parquet.Type, _ int32) bool ( IntLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( IntLogicalType) IsNested() bool ( IntLogicalType) IsNone() bool ( IntLogicalType) IsSerialized() bool ( IntLogicalType) IsSigned() bool ( IntLogicalType) IsValid() bool ( IntLogicalType) MarshalJSON() ([]byte, error) ( IntLogicalType) SortOrder() SortOrder ( IntLogicalType) String() string ( IntLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) IntLogicalType : LogicalType IntLogicalType : github.com/goccy/go-json.Marshaler IntLogicalType : database/sql/driver.Validator IntLogicalType : encoding/json.Marshaler IntLogicalType : expvar.Var IntLogicalType : fmt.Stringer
JSONLogicalType represents a byte array column which is to be interpreted as a JSON string. ( JSONLogicalType) Equals(rhs LogicalType) bool ( JSONLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( JSONLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( JSONLogicalType) IsNested() bool ( JSONLogicalType) IsNone() bool ( JSONLogicalType) IsSerialized() bool ( JSONLogicalType) IsValid() bool ( JSONLogicalType) MarshalJSON() ([]byte, error) ( JSONLogicalType) SortOrder() SortOrder ( JSONLogicalType) String() string ( JSONLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) JSONLogicalType : LogicalType JSONLogicalType : github.com/goccy/go-json.Marshaler JSONLogicalType : database/sql/driver.Validator JSONLogicalType : encoding/json.Marshaler JSONLogicalType : expvar.Var JSONLogicalType : fmt.Stringer
ListLogicalType is used for columns which are themselves nested lists ( ListLogicalType) Equals(rhs LogicalType) bool ( ListLogicalType) IsApplicable(parquet.Type, int32) bool ( ListLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool ( ListLogicalType) IsNested() bool ( ListLogicalType) IsNone() bool ( ListLogicalType) IsSerialized() bool ( ListLogicalType) IsValid() bool ( ListLogicalType) MarshalJSON() ([]byte, error) ( ListLogicalType) SortOrder() SortOrder ( ListLogicalType) String() string ( ListLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) ListLogicalType : LogicalType ListLogicalType : github.com/goccy/go-json.Marshaler ListLogicalType : database/sql/driver.Validator ListLogicalType : encoding/json.Marshaler ListLogicalType : expvar.Var ListLogicalType : fmt.Stringer
LogicalType is the descriptor that defines the usage of a physical primitive type in the schema, such as an Interval, Date, etc. Returns true if the logical types are the same Returns true if this logical type can be used with the provided physical type Returns true if the specified ConvertedType is compatible with this logical type Returns true if a nested type like List or Map Returns true if it is NoType Returns true if this type can be serialized, ie: not Unknown/NoType/Interval Returns true if not NoLogicalType Returns the default stat sort order for this logical type returns a string representation of the Logical Type Return the equivalent ConvertedType for legacy Parquet systems BSONLogicalType DateLogicalType DecimalLogicalType EnumLogicalType Float16LogicalType IntervalLogicalType IntLogicalType JSONLogicalType ListLogicalType MapLogicalType NoLogicalType NullLogicalType StringLogicalType TemporalLogicalType (interface) TimeLogicalType TimestampLogicalType UnknownLogicalType UUIDLogicalType VariantLogicalType LogicalType : database/sql/driver.Validator LogicalType : expvar.Var LogicalType : fmt.Stringer func NewDecimalLogicalType(precision int32, scale int32) LogicalType func NewIntLogicalType(bitWidth int8, signed bool) LogicalType func NewListLogicalType() LogicalType func NewTimeLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func NewTimestampLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func NewTimestampLogicalTypeForce(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func NewTimestampLogicalTypeWithOpts(opts ...TimestampOpt) LogicalType func (*Column).LogicalType() LogicalType func ConvertedType.ToLogicalType(convertedDecimal DecimalMetadata) LogicalType func Node.LogicalType() LogicalType func github.com/apache/arrow-go/v18/arrow/extensions.(*JSONType).ParquetLogicalType() LogicalType func github.com/apache/arrow-go/v18/arrow/extensions.(*UUIDType).ParquetLogicalType() LogicalType func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantType).ParquetLogicalType() LogicalType func GetLogicalSortOrder(logical LogicalType, primitive format.Type) SortOrder func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error) func NewPrimitiveNodeLogical(name string, repetition parquet.Repetition, logicalType LogicalType, physicalType parquet.Type, typeLen int, id int32) (*PrimitiveNode, error) func BSONLogicalType.Equals(rhs LogicalType) bool func DateLogicalType.Equals(rhs LogicalType) bool func DecimalLogicalType.Equals(rhs LogicalType) bool func EnumLogicalType.Equals(rhs LogicalType) bool func Float16LogicalType.Equals(rhs LogicalType) bool func IntervalLogicalType.Equals(rhs LogicalType) bool func IntLogicalType.Equals(rhs LogicalType) bool func JSONLogicalType.Equals(rhs LogicalType) bool func ListLogicalType.Equals(rhs LogicalType) bool func LogicalType.Equals(LogicalType) bool func MapLogicalType.Equals(rhs LogicalType) bool func NoLogicalType.Equals(rhs LogicalType) bool func NullLogicalType.Equals(rhs LogicalType) bool func StringLogicalType.Equals(rhs LogicalType) bool func TemporalLogicalType.Equals(LogicalType) bool func TimeLogicalType.Equals(rhs LogicalType) bool func TimestampLogicalType.Equals(rhs LogicalType) bool func UnknownLogicalType.Equals(rhs LogicalType) bool func UUIDLogicalType.Equals(rhs LogicalType) bool func VariantLogicalType.Equals(rhs LogicalType) bool
MapLogicalType represents a mapped type ( MapLogicalType) Equals(rhs LogicalType) bool ( MapLogicalType) IsApplicable(parquet.Type, int32) bool ( MapLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool ( MapLogicalType) IsNested() bool ( MapLogicalType) IsNone() bool ( MapLogicalType) IsSerialized() bool ( MapLogicalType) IsValid() bool ( MapLogicalType) MarshalJSON() ([]byte, error) ( MapLogicalType) SortOrder() SortOrder ( MapLogicalType) String() string ( MapLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) MapLogicalType : LogicalType MapLogicalType : github.com/goccy/go-json.Marshaler MapLogicalType : database/sql/driver.Validator MapLogicalType : encoding/json.Marshaler MapLogicalType : expvar.Var MapLogicalType : fmt.Stringer
Node is the interface for both Group and Primitive Nodes. A logical schema type has a name, repetition level, and optionally a logical type (converted type is the deprecated version of the logical type concept, which is maintained for forward compatibility) ( Node) ConvertedType() ConvertedType ( Node) Equals(Node) bool ( Node) FieldID() int32 ( Node) LogicalType() LogicalType ( Node) Name() string ( Node) Parent() Node ( Node) Path() string ( Node) RepetitionType() parquet.Repetition ( Node) SetParent(Node) ( Node) Type() NodeType ( Node) Visit(v Visitor) *GroupNode *PrimitiveNode Node : github.com/polarsignals/frostdb/query/logicalplan.Named func FromParquet(elems []*format.SchemaElement) (Node, error) func Must(n Node, err error) Node func (*Column).SchemaNode() Node func (*GroupNode).Field(i int) Node func Node.Parent() Node func (*Schema).ColumnRoot(i int) Node func ColumnPathFromNode(n Node) parquet.ColumnPath func ListOf(n Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error) func ListOfWithName(listName string, element Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error) func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error) func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error) func Must(n Node, err error) Node func MustGroup(n Node, err error) *GroupNode func MustPrimitive(n Node, err error) *PrimitiveNode func PrintSchema(n Node, w io.Writer, indentWidth int) func (*GroupNode).Equals(rhs Node) bool func (*GroupNode).FieldIndexByField(n Node) int func Node.Equals(Node) bool func Node.SetParent(Node) func (*PrimitiveNode).Equals(rhs Node) bool func (*Schema).ColumnIndexByNode(n Node) int func Visitor.VisitPost(Node) func Visitor.VisitPre(Node) bool
NodeType describes whether the Node is a Primitive or Group node func Node.Type() NodeType const Group const Primitive
( NoLogicalType) Equals(rhs LogicalType) bool ( NoLogicalType) IsApplicable(parquet.Type, int32) bool ( NoLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( NoLogicalType) IsNested() bool ( NoLogicalType) IsNone() bool ( NoLogicalType) IsSerialized() bool ( NoLogicalType) IsValid() bool ( NoLogicalType) MarshalJSON() ([]byte, error) ( NoLogicalType) SortOrder() SortOrder ( NoLogicalType) String() string ( NoLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) NoLogicalType : LogicalType NoLogicalType : github.com/goccy/go-json.Marshaler NoLogicalType : database/sql/driver.Validator NoLogicalType : encoding/json.Marshaler NoLogicalType : expvar.Var NoLogicalType : fmt.Stringer
( NullLogicalType) Equals(rhs LogicalType) bool ( NullLogicalType) IsApplicable(parquet.Type, int32) bool ( NullLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( NullLogicalType) IsNested() bool ( NullLogicalType) IsNone() bool ( NullLogicalType) IsSerialized() bool ( NullLogicalType) IsValid() bool ( NullLogicalType) MarshalJSON() ([]byte, error) ( NullLogicalType) SortOrder() SortOrder ( NullLogicalType) String() string ( NullLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) NullLogicalType : LogicalType NullLogicalType : github.com/goccy/go-json.Marshaler NullLogicalType : database/sql/driver.Validator NullLogicalType : encoding/json.Marshaler NullLogicalType : expvar.Var NullLogicalType : fmt.Stringer
A PrimitiveNode is a type that is one of the primitive Parquet storage types. In addition to the other type metadata (name, repetition level, logical type), also has the physical storage type and their type-specific metadata (byte width, decimal parameters) ColumnOrder parquet.ColumnOrder (*PrimitiveNode) ConvertedType() ConvertedType DecimalMetadata returns the current metadata for the node. If not a decimal typed column, the return should have IsSet == false. Equals returns true if both nodes are primitive nodes with the same physical and converted/logical types. (*PrimitiveNode) FieldID() int32 (*PrimitiveNode) LogicalType() LogicalType (*PrimitiveNode) Name() string (*PrimitiveNode) Parent() Node (*PrimitiveNode) Path() string PhysicalType returns the proper Physical parquet.Type primitive that is used to store the values in this column. (*PrimitiveNode) RepetitionType() parquet.Repetition (*PrimitiveNode) SetParent(p Node) SetTypeLength will change the type length of the node, has no effect if the physical type is not FixedLength Byte Array (*PrimitiveNode) Type() NodeType TypeLength will be -1 if not a FixedLenByteArray column, otherwise will be the length of the FixedLen Byte Array Visit is for implementing a Visitor pattern handler to walk a schema's tree. One example is the Schema Printer which walks the tree to print out the schema in order. *PrimitiveNode : Node *PrimitiveNode : github.com/polarsignals/frostdb/query/logicalplan.Named func MustPrimitive(n Node, err error) *PrimitiveNode func NewBooleanNode(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewByteArrayNode(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewFixedLenByteArrayNode(name string, rep parquet.Repetition, length int32, fieldID int32) *PrimitiveNode func NewFloat32Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewFloat64Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewInt32Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewInt64Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewInt96Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode func NewPrimitiveNode(name string, repetition parquet.Repetition, typ parquet.Type, fieldID, typeLength int32) (*PrimitiveNode, error) func NewPrimitiveNodeConverted(name string, repetition parquet.Repetition, typ parquet.Type, converted ConvertedType, typeLen, precision, scale int, id int32) (*PrimitiveNode, error) func NewPrimitiveNodeLogical(name string, repetition parquet.Repetition, logicalType LogicalType, physicalType parquet.Type, typeLen int, id int32) (*PrimitiveNode, error) func PrimitiveNodeFromThrift(elem *format.SchemaElement) (*PrimitiveNode, error) func NewColumn(n *PrimitiveNode, maxDefinitionLvl, maxRepetitionLvl int16) *Column
Schema is the container for the converted Parquet schema with a computed information from the schema analysis needed for file reading * Column index to Node * Max repetition / definition levels for each primitive node The ColumnDescriptor objects produced by this class can be used to assist in the reconstruction of fully materialized data structures from the repetition-definition level encoding of nested data Column returns the (0-indexed) column of the provided index. ColumnIndexByName looks up the column by it's full dot separated node path. If there are multiple columns that match, it returns the first one. Returns -1 if not found. ColumnIndexByNode returns the index of the column represented by this node. Returns -1 if not found. ColumnRoot returns the root node of a given column if it is under a nested group node, providing that root group node. Columns returns an iterator over the leaf columns of the schema Equals returns true as long as the leaf columns are equal, doesn't take into account the groups and only checks whether the schemas are compatible at the physical storage level. HasRepeatedFields returns true if any node in the schema has a repeated field type. NumColumns returns the number of leaf nodes that are the actual primitive columns in this schema. Root returns the group node that is the root of this schema (*Schema) String() string UpdateColumnOrders must get a slice that is the same length as the number of leaf columns and is used to update the schema metadata Column Orders. len(orders) must equal s.NumColumns() *Schema : expvar.Var *Schema : fmt.Stringer func NewSchema(root *GroupNode) *Schema func NewSchemaFromStruct(obj interface{}) (sc *Schema, err error) func NewStructFromSchema(sc *Schema) (t reflect.Type, err error) func (*Schema).Equals(rhs *Schema) bool
SortOrder mirrors the parquet.thrift sort order type func DefaultSortOrder(primitive format.Type) SortOrder func GetLogicalSortOrder(logical LogicalType, primitive format.Type) SortOrder func GetSortOrder(convert ConvertedType, primitive format.Type) SortOrder func BSONLogicalType.SortOrder() SortOrder func (*Column).SortOrder() SortOrder func DateLogicalType.SortOrder() SortOrder func DecimalLogicalType.SortOrder() SortOrder func EnumLogicalType.SortOrder() SortOrder func Float16LogicalType.SortOrder() SortOrder func IntervalLogicalType.SortOrder() SortOrder func IntLogicalType.SortOrder() SortOrder func JSONLogicalType.SortOrder() SortOrder func ListLogicalType.SortOrder() SortOrder func LogicalType.SortOrder() SortOrder func MapLogicalType.SortOrder() SortOrder func NoLogicalType.SortOrder() SortOrder func NullLogicalType.SortOrder() SortOrder func StringLogicalType.SortOrder() SortOrder func TemporalLogicalType.SortOrder() SortOrder func TimeLogicalType.SortOrder() SortOrder func TimestampLogicalType.SortOrder() SortOrder func UnknownLogicalType.SortOrder() SortOrder func UUIDLogicalType.SortOrder() SortOrder func VariantLogicalType.SortOrder() SortOrder const SortSIGNED const SortUNKNOWN const SortUNSIGNED
StringLogicalType is a UTF8 string, only usable with ByteArray and FixedLenByteArray ( StringLogicalType) Equals(rhs LogicalType) bool ( StringLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( StringLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool ( StringLogicalType) IsNested() bool ( StringLogicalType) IsNone() bool ( StringLogicalType) IsSerialized() bool ( StringLogicalType) IsValid() bool ( StringLogicalType) MarshalJSON() ([]byte, error) ( StringLogicalType) SortOrder() SortOrder ( StringLogicalType) String() string ( StringLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) StringLogicalType : LogicalType StringLogicalType : github.com/goccy/go-json.Marshaler StringLogicalType : database/sql/driver.Validator StringLogicalType : encoding/json.Marshaler StringLogicalType : expvar.Var StringLogicalType : fmt.Stringer
TemporalLogicalType is a smaller interface for Time based logical types like Time / Timestamp Returns true if the logical types are the same ( TemporalLogicalType) IsAdjustedToUTC() bool Returns true if this logical type can be used with the provided physical type Returns true if the specified ConvertedType is compatible with this logical type Returns true if a nested type like List or Map Returns true if it is NoType Returns true if this type can be serialized, ie: not Unknown/NoType/Interval Returns true if not NoLogicalType Returns the default stat sort order for this logical type returns a string representation of the Logical Type ( TemporalLogicalType) TimeUnit() TimeUnitType Return the equivalent ConvertedType for legacy Parquet systems TimeLogicalType TimestampLogicalType TemporalLogicalType : LogicalType TemporalLogicalType : database/sql/driver.Validator TemporalLogicalType : expvar.Var TemporalLogicalType : fmt.Stringer
TimeLogicalType is a time type without a date and must be an int32 for milliseconds, or an int64 for micro or nano seconds. ( TimeLogicalType) Equals(rhs LogicalType) bool ( TimeLogicalType) IsAdjustedToUTC() bool ( TimeLogicalType) IsApplicable(typ parquet.Type, _ int32) bool ( TimeLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( TimeLogicalType) IsNested() bool ( TimeLogicalType) IsNone() bool ( TimeLogicalType) IsSerialized() bool ( TimeLogicalType) IsValid() bool ( TimeLogicalType) MarshalJSON() ([]byte, error) ( TimeLogicalType) SortOrder() SortOrder ( TimeLogicalType) String() string ( TimeLogicalType) TimeUnit() TimeUnitType ( TimeLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) TimeLogicalType : LogicalType TimeLogicalType : TemporalLogicalType TimeLogicalType : github.com/goccy/go-json.Marshaler TimeLogicalType : database/sql/driver.Validator TimeLogicalType : encoding/json.Marshaler TimeLogicalType : expvar.Var TimeLogicalType : fmt.Stringer
TimestampLogicalType represents an int64 number that can be decoded into a year, month, day, hour, minute, second, and subsecond ( TimestampLogicalType) Equals(rhs LogicalType) bool ( TimestampLogicalType) IsAdjustedToUTC() bool ( TimestampLogicalType) IsApplicable(t parquet.Type, _ int32) bool ( TimestampLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( TimestampLogicalType) IsFromConvertedType() bool ( TimestampLogicalType) IsNested() bool ( TimestampLogicalType) IsNone() bool ( TimestampLogicalType) IsSerialized() bool ( TimestampLogicalType) IsValid() bool ( TimestampLogicalType) MarshalJSON() ([]byte, error) ( TimestampLogicalType) SortOrder() SortOrder ( TimestampLogicalType) String() string ( TimestampLogicalType) TimeUnit() TimeUnitType ( TimestampLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) TimestampLogicalType : LogicalType TimestampLogicalType : TemporalLogicalType TimestampLogicalType : github.com/goccy/go-json.Marshaler TimestampLogicalType : database/sql/driver.Validator TimestampLogicalType : encoding/json.Marshaler TimestampLogicalType : expvar.Var TimestampLogicalType : fmt.Stringer
TimestampOpt options used with New Timestamp Logical Type func WithTSForceConverted() TimestampOpt func WithTSFromConverted() TimestampOpt func WithTSIsAdjustedToUTC() TimestampOpt func WithTSTimeUnitType(unit TimeUnitType) TimestampOpt func NewTimestampLogicalTypeWithOpts(opts ...TimestampOpt) LogicalType
TimeUnitType is an enum for denoting whether a time based logical type is using milliseconds, microseconds or nanoseconds. func TemporalLogicalType.TimeUnit() TimeUnitType func TimeLogicalType.TimeUnit() TimeUnitType func TimestampLogicalType.TimeUnit() TimeUnitType func NewTimeLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func NewTimestampLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func NewTimestampLogicalTypeForce(isAdjustedToUTC bool, unit TimeUnitType) LogicalType func WithTSTimeUnitType(unit TimeUnitType) TimestampOpt const TimeUnitMicros const TimeUnitMillis const TimeUnitNanos const TimeUnitUnknown
UnknownLogicalType is a type that is essentially a placeholder for when we don't know the type. ( UnknownLogicalType) Equals(rhs LogicalType) bool ( UnknownLogicalType) IsApplicable(parquet.Type, int32) bool ( UnknownLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( UnknownLogicalType) IsNested() bool ( UnknownLogicalType) IsNone() bool ( UnknownLogicalType) IsSerialized() bool ( UnknownLogicalType) IsValid() bool ( UnknownLogicalType) MarshalJSON() ([]byte, error) ( UnknownLogicalType) SortOrder() SortOrder ( UnknownLogicalType) String() string ( UnknownLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) UnknownLogicalType : LogicalType UnknownLogicalType : github.com/goccy/go-json.Marshaler UnknownLogicalType : database/sql/driver.Validator UnknownLogicalType : encoding/json.Marshaler UnknownLogicalType : expvar.Var UnknownLogicalType : fmt.Stringer
UUIDLogicalType can only be used with a FixedLength byte array column that is exactly 16 bytes long ( UUIDLogicalType) Equals(rhs LogicalType) bool ( UUIDLogicalType) IsApplicable(t parquet.Type, tlen int32) bool ( UUIDLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool ( UUIDLogicalType) IsNested() bool ( UUIDLogicalType) IsNone() bool ( UUIDLogicalType) IsSerialized() bool ( UUIDLogicalType) IsValid() bool ( UUIDLogicalType) MarshalJSON() ([]byte, error) ( UUIDLogicalType) SortOrder() SortOrder ( UUIDLogicalType) String() string ( UUIDLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) UUIDLogicalType : LogicalType UUIDLogicalType : github.com/goccy/go-json.Marshaler UUIDLogicalType : database/sql/driver.Validator UUIDLogicalType : encoding/json.Marshaler UUIDLogicalType : expvar.Var UUIDLogicalType : fmt.Stringer
( VariantLogicalType) Equals(rhs LogicalType) bool ( VariantLogicalType) IsApplicable(parquet.Type, int32) bool ( VariantLogicalType) IsCompatible(ct ConvertedType, _ DecimalMetadata) bool ( VariantLogicalType) IsNested() bool ( VariantLogicalType) IsNone() bool ( VariantLogicalType) IsSerialized() bool ( VariantLogicalType) IsValid() bool ( VariantLogicalType) MarshalJSON() ([]byte, error) ( VariantLogicalType) SortOrder() SortOrder ( VariantLogicalType) String() string ( VariantLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata) VariantLogicalType : LogicalType VariantLogicalType : github.com/goccy/go-json.Marshaler VariantLogicalType : database/sql/driver.Validator VariantLogicalType : encoding/json.Marshaler VariantLogicalType : expvar.Var VariantLogicalType : fmt.Stringer
Visitor is an interface for creating functionality to walk the schema tree. A visitor can be passed to the Visit function of a Node in order to walk the tree. VisitPre is called the first time a node is encountered. If it is a group node, the return is checked and if it is false, the children will be skipped. VisitPost is called after visiting any children ( Visitor) VisitPost(Node) ( Visitor) VisitPre(Node) bool func (*GroupNode).Visit(v Visitor) func Node.Visit(v Visitor) func (*PrimitiveNode).Visit(v Visitor)
Package-Level Functions (total 44)
ColumnPathFromNode walks the parents of the given node to construct it's column path
DefaultSortOrder returns the default stat sort order for the given physical type
FromParquet converts a slice of thrift Schema Elements to the correct node type
GetLogicalSortOrder returns the default sort order for this logical type or falls back to the default sort order for the physical type if not valid
GetSortOrder defaults to the sort order based on the physical type if convert is ConvertedTypes.None, otherwise determines the sort order by the converted type.
ListOf is a convenience helper function to create a properly structured list structure according to the Parquet Spec. <list-repetition> group <name> (LIST) { repeated group list { <element-repetition> <element-type> element; } } <list-repetition> can only be optional or required. <element-repetition> can only be optional or required.
ListOf is a convenience helper function to create a properly structured list structure according to the Parquet Spec. <list-repetition> group <name> (LIST) { repeated group list { <element-repetition> <element-type> element; } } <list-repetition> can only be optional or required. <element-repetition> can only be optional or required.
MapOf is a convenience helper function to create a properly structured parquet map node setup according to the Parquet Spec. <map-repetition> group <name> (MAP) { repeated group key_value { required <key-type> key; <value-repetition> <value-type> value; } } key node will be renamed to "key", value node if not nil will be renamed to "value" <map-repetition> must be only optional or required. panics if repeated is passed. the key node *must* be required repetition. panics if optional or repeated value node can be nil (omitted) or have a repetition of required or optional *only*.
Must is a convenience function for the NewNode functions that return a Node and an error, panic'ing if err != nil or returning the node
MustGroup is like Must, except it casts the node to a *GroupNode, which will panic if it is a primitive node.
MustPrimitive is like Must except it casts the node to *PrimitiveNode which will panic if it is a group node.
NewBooleanNode is a convenience factory for constructing an Boolean Primitive Node
NewByteArrayNode is a convenience factory for constructing an Byte Array Primitive Node
NewColumn returns a new column object for the given node with the provided maximum definition and repetition levels.
NewDecimalLogicalType returns a Decimal logical type with the given precision and scale. Panics if precision < 1 or scale is not in the range (0, precision)
NewFixedLenByteArrayNode is a convenience factory for constructing an Fixed Length Byte Array Primitive Node of the given length
NewFloat32Node is a convenience factory for constructing an Float Primitive Node
NewFloat64Node is a convenience factory for constructing an Double Primitive Node
NewGroupNode constructs a new group node with the provided fields, but with converted type None and No Logical Type
NewGroupNodeConverted constructs a group node with the provided fields and converted type, determining the logical type from that converted type.
NewGroupNodeLogical constructs a group node with the provided fields and logical type, determining the converted type from the provided logical type.
NewInt32Node is a convenience factory for constructing an Int32 Primitive Node
NewInt64Node is a convenience factory for constructing an Int64 Primitive Node
NewInt96Node is a convenience factory for constructing an Int96 Primitive Node
NewIntLogicalType creates an integer logical type of the desired bitwidth and whether it is signed or not. Bit width must be exactly 8, 16, 32 or 64 for an integer logical type
NewPrimitiveNode constructs a primitive node with the ConvertedType of None and no logical type. Use NewPrimitiveNodeLogical and NewPrimitiveNodeConverted to specify the logical or converted type.
NewPrimitiveNodeConverted constructs a primitive node from the given physical type and converted type, determining the logical type from the converted type.
NewPrimitiveNodeLogical constructs a Primitive node using the provided logical type for a given physical type and typelength.
NewSchema constructs a new Schema object from a root group node. Any fields with a field-id of -1 will be given an appropriate field number based on their order.
NewSchemaFromStruct generates a schema from an object type via reflection of the type and reading struct tags for "parquet". # Rules Everything defaults to Required repetition, unless otherwise specified. Pointer types become Optional repetition. Arrays and Slices become logical List types unless using the tag `repetition=repeated`. A length specified byte field (like [5]byte) becomes a fixed_len_byte_array of that length unless otherwise specified by tags. string and []byte both become ByteArray unless otherwise specified. Integer types will default to having a logical type of the appropriate bit width and signedness rather than having no logical type, ie: an int8 will become an int32 node with logical type Int(bitWidth=8, signed=true). Structs will become group nodes with the fields of the struct as the fields of the group, recursively creating the nodes. maps will become appropriate Map structures in the schema of the defined key and values. # Available Tags name: by default the node will have the same name as the field, this tag let's you specify a name type: Specify the physical type instead of using the field type length: specify the type length of the node, only relevant for fixed_len_byte_array scale: specify the scale for a decimal field precision: specify the precision for a decimal field fieldid: specify the field ID for that node, defaults to -1 which means it is not set in the parquet file. repetition: specify the repetition as something other than what is determined by the type converted: specify the Converted Type of the field logical: specify the logical type of the field, if using decimal then the scale and precision will be determined by the precision and scale fields, or by the logical.precision / logical.scale fields with the logical. prefixed versions taking precedence. For Time or Timestamp logical types, use logical.unit=<millis|micros|nanos> and logical.isadjustedutc=<true|false> to set those. Unit is required isadjustedutc defaults to true. For Integer logical type, use logical.bitwidth and logical.signed to specify those values, with bitwidth being required, and signed defaulting to true. All tags other than name can use a prefix of "key<tagname>=<value>" to refer to the type of the key for a map and "value<tagname>=<value>" to refer to the value type of a map or the element of a list (such as the type of a slice)
NewStructFromSchema generates a struct type as a reflect.Type from the schema by using the appropriate physical types and making things either pointers or slices based on whether they are repeated/optional/required. It does not use the logical or converted types to change the physical storage so that it is more efficient to use the resulting type for reading without having to do conversions. It will use maps for map types and slices for list types, but otherwise ignores the converted and logical types of the nodes. Group nodes that are not List or Map will be nested structs.
NewTimeLogicalType returns a time type of the given unit.
NewTimestampLogicalType returns a logical timestamp type with "forceConverted" set to false
NewTimestampLogicalTypeForce returns a timestamp logical type with "forceConverted" set to true
NewTimestampLogicalTypeWithOpts creates a new TimestampLogicalType with the provided options. TimestampType Unit defaults to milliseconds (TimeUnitMillis)
PrintSchema writes a string representation of the tree to w using the indent width provided.
ToThrift converts a GroupNode to a slice of SchemaElements which is used for thrift serialization.
WithTSForceConverted enable force converted mode
WithTSFromConverted enable the timestamp logical type to be constructed from a converted type.
WithTSIsAdjustedToUTC sets the IsAdjustedToUTC field of the timestamp type.
WithTSTimeUnitType sets the time unit for the timestamp type
Package-Level Variables (only one)
ConvertedTypes is a struct containing the constants for the types to make it easy to reference them while making it clear what they are
Package-Level Constants (total 9)
the available constants for NodeType
the available constants for NodeType
Constants for the Stat sort order definitions
Constants for the Stat sort order definitions
Constants for the Stat sort order definitions
Constants for the TimeUnitType
Constants for the TimeUnitType
Constants for the TimeUnitType
Constants for the TimeUnitType