package schema
Import Path
github.com/apache/arrow-go/v18/parquet/schema (on go.dev)
Dependency Relation
imports 16 packages, and imported by one package
Involved Source Files
column.go
converted_types.go
helpers.go
logical_types.go
node.go
reflection.go
Package schema provides types and functions for manipulating and building parquet
file schemas.
Some of the utilities provided include building a schema using Struct Tags
on a struct type, getting Column Paths from a node, and dealing with the
converted and logical types for Parquet.
Logical types specify ways to interpret the primitive types allowing the
number of primitive types to be smaller and reuse efficient encodings.
For instance a "string" is just a ByteArray column with a UTF-8 annotation
or "String Logical Type".
For more information about Logical and Converted Types, check:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
Package-Level Type Names (total 33)
BSONLogicalType represents a binary JSON string in the byte array
( BSONLogicalType) Equals(rhs LogicalType) bool
( BSONLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( BSONLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( BSONLogicalType) IsNested() bool
( BSONLogicalType) IsNone() bool
( BSONLogicalType) IsSerialized() bool
( BSONLogicalType) IsValid() bool
( BSONLogicalType) MarshalJSON() ([]byte, error)
( BSONLogicalType) SortOrder() SortOrder
( BSONLogicalType) String() string
( BSONLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
BSONLogicalType : LogicalType
BSONLogicalType : github.com/goccy/go-json.Marshaler
BSONLogicalType : database/sql/driver.Validator
BSONLogicalType : encoding/json.Marshaler
BSONLogicalType : expvar.Var
BSONLogicalType : fmt.Stringer
Column encapsulates the information necessary to interpret primitive
column data in the context of a particular schema. We have to examine
the node structure of a column's path to the root in the schema tree
to be able to reassemble the nested structure from the repetition and
definition levels.
(*Column) ColumnOrder() parquet.ColumnOrder
ColumnPath returns the full path to this column from the root of the schema
(*Column) ConvertedType() ConvertedType
Equals will return true if the rhs Column has the same Max Repetition and Definition levels
along with having the same node definition.
(*Column) LogicalType() LogicalType
(*Column) MaxDefinitionLevel() int16
(*Column) MaxRepetitionLevel() int16
Name is the column's name
Path is equivalent to ColumnPath().String() returning the dot-string version of the path
(*Column) PhysicalType() parquet.Type
SchemaNode returns the underlying Node in the schema tree for this column.
SortOrder returns the sort order of this column's statistics based on the
Logical and Converted types.
(*Column) String() string
TypeLength is -1 if not a FixedLenByteArray, otherwise it is the length of elements in the column
*Column : github.com/polarsignals/frostdb/query/logicalplan.Named
*Column : expvar.Var
*Column : fmt.Stringer
func NewColumn(n *PrimitiveNode, maxDefinitionLvl, maxRepetitionLvl int16) *Column
func (*Schema).Column(i int) *Column
func (*Column).Equals(rhs *Column) bool
ConvertedType corresponds to the ConvertedType in the parquet.Thrift,
with added values of None and NA for handling when these values are not
set in the metadata
( ConvertedType) String() string
ToLogicalType returns the correct LogicalType for the given ConvertedType, using the decimal
metadata provided to define the precision/scale if necessary
ConvertedType : expvar.Var
ConvertedType : fmt.Stringer
func BSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func (*Column).ConvertedType() ConvertedType
func DateLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func DecimalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func EnumLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func Float16LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func IntervalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func IntLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func JSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func ListLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func MapLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func Node.ConvertedType() ConvertedType
func NoLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func NullLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func StringLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TemporalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TimeLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TimestampLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func UnknownLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func UUIDLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func VariantLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func GetSortOrder(convert ConvertedType, primitive format.Type) SortOrder
func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error)
func NewPrimitiveNodeConverted(name string, repetition parquet.Repetition, typ parquet.Type, converted ConvertedType, typeLen, precision, scale int, id int32) (*PrimitiveNode, error)
func BSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func DateLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func DecimalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func EnumLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func Float16LogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func IntervalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func IntLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func JSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func ListLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func LogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool
func MapLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func NoLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func NullLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func StringLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func TemporalLogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool
func TimeLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func TimestampLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func UnknownLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func UUIDLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func VariantLogicalType.IsCompatible(ct ConvertedType, _ DecimalMetadata) bool
DateLogicalType is an int32 representing the number of days since the Unix Epoch
1 January 1970
( DateLogicalType) Equals(rhs LogicalType) bool
( DateLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( DateLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool
( DateLogicalType) IsNested() bool
( DateLogicalType) IsNone() bool
( DateLogicalType) IsSerialized() bool
( DateLogicalType) IsValid() bool
( DateLogicalType) MarshalJSON() ([]byte, error)
( DateLogicalType) SortOrder() SortOrder
( DateLogicalType) String() string
( DateLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
DateLogicalType : LogicalType
DateLogicalType : github.com/goccy/go-json.Marshaler
DateLogicalType : database/sql/driver.Validator
DateLogicalType : encoding/json.Marshaler
DateLogicalType : expvar.Var
DateLogicalType : fmt.Stringer
DecimalLogicalType is used to represent a decimal value of a given
precision and scale
( DecimalLogicalType) Equals(rhs LogicalType) bool
( DecimalLogicalType) IsApplicable(typ parquet.Type, tlen int32) bool
( DecimalLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( DecimalLogicalType) IsNested() bool
( DecimalLogicalType) IsNone() bool
( DecimalLogicalType) IsSerialized() bool
( DecimalLogicalType) IsValid() bool
( DecimalLogicalType) MarshalJSON() ([]byte, error)
( DecimalLogicalType) Precision() int32
( DecimalLogicalType) Scale() int32
( DecimalLogicalType) SortOrder() SortOrder
( DecimalLogicalType) String() string
( DecimalLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
DecimalLogicalType : LogicalType
DecimalLogicalType : github.com/goccy/go-json.Marshaler
DecimalLogicalType : database/sql/driver.Validator
DecimalLogicalType : encoding/json.Marshaler
DecimalLogicalType : expvar.Var
DecimalLogicalType : fmt.Stringer
DecimalMetadata is a struct for managing scale and precision information between
converted and logical types.
IsSet bool
Precision int32
Scale int32
func BSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func DateLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func DecimalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func EnumLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func Float16LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func IntervalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func IntLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func JSONLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func ListLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func LogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func MapLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func NoLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func NullLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func (*PrimitiveNode).DecimalMetadata() DecimalMetadata
func StringLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TemporalLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TimeLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func TimestampLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func UnknownLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func UUIDLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func VariantLogicalType.ToConvertedType() (ConvertedType, DecimalMetadata)
func BSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func ConvertedType.ToLogicalType(convertedDecimal DecimalMetadata) LogicalType
func DateLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func DecimalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func EnumLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func Float16LogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func IntervalLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func IntLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func JSONLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func ListLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func LogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool
func MapLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func NoLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func NullLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func StringLogicalType.IsCompatible(t ConvertedType, dec DecimalMetadata) bool
func TemporalLogicalType.IsCompatible(ConvertedType, DecimalMetadata) bool
func TimeLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func TimestampLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func UnknownLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func UUIDLogicalType.IsCompatible(c ConvertedType, dec DecimalMetadata) bool
func VariantLogicalType.IsCompatible(ct ConvertedType, _ DecimalMetadata) bool
EnumLogicalType is for representing an enum, which should be a byte array type
( EnumLogicalType) Equals(rhs LogicalType) bool
( EnumLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( EnumLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool
( EnumLogicalType) IsNested() bool
( EnumLogicalType) IsNone() bool
( EnumLogicalType) IsSerialized() bool
( EnumLogicalType) IsValid() bool
( EnumLogicalType) MarshalJSON() ([]byte, error)
( EnumLogicalType) SortOrder() SortOrder
( EnumLogicalType) String() string
( EnumLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
EnumLogicalType : LogicalType
EnumLogicalType : github.com/goccy/go-json.Marshaler
EnumLogicalType : database/sql/driver.Validator
EnumLogicalType : encoding/json.Marshaler
EnumLogicalType : expvar.Var
EnumLogicalType : fmt.Stringer
FieldList is an alias for a slice of Nodes
Len is equivalent to len(fieldlist)
func GroupNodeFromThrift(elem *format.SchemaElement, fields FieldList) (*GroupNode, error)
func NewGroupNode(name string, repetition parquet.Repetition, fields FieldList, fieldID int32) (*GroupNode, error)
func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error)
func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error)
Float16LogicalType can only be used with a FixedLength byte array column
that is exactly 2 bytes long
( Float16LogicalType) Equals(rhs LogicalType) bool
( Float16LogicalType) IsApplicable(t parquet.Type, tlen int32) bool
( Float16LogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( Float16LogicalType) IsNested() bool
( Float16LogicalType) IsNone() bool
( Float16LogicalType) IsSerialized() bool
( Float16LogicalType) IsValid() bool
( Float16LogicalType) MarshalJSON() ([]byte, error)
( Float16LogicalType) SortOrder() SortOrder
( Float16LogicalType) String() string
( Float16LogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
Float16LogicalType : LogicalType
Float16LogicalType : github.com/goccy/go-json.Marshaler
Float16LogicalType : database/sql/driver.Validator
Float16LogicalType : encoding/json.Marshaler
Float16LogicalType : expvar.Var
Float16LogicalType : fmt.Stringer
GroupNode is for managing nested nodes like List, Map, etc.
(*GroupNode) ConvertedType() ConvertedType
Equals will compare this node to the provided node and only return true if
this node and all of it's children are the same as the passed in node and its
children.
Field returns the node in the field list which is of the provided (0-based) index
(*GroupNode) FieldID() int32
FieldIndexByField looks up the index child of this node. Returns -1
if n isn't a child of this group
FieldIndexByName provides the index for the field of the given name. Returns
-1 if not found.
If there are more than one field of this name, it returns the index for the first one.
HasRepeatedFields returns true if any of the children of this node have
Repeated as its repetition type.
This is recursive and will check the children of any group nodes that are children.
(*GroupNode) LogicalType() LogicalType
(*GroupNode) Name() string
NumFields returns the number of direct child fields for this group node
(*GroupNode) Parent() Node
(*GroupNode) Path() string
(*GroupNode) RepetitionType() parquet.Repetition
(*GroupNode) SetParent(p Node)
(*GroupNode) Type() NodeType
Visit is for implementing a Visitor pattern handler to walk a schema's tree. One
example is the Schema Printer which walks the tree to print out the schema in order.
*GroupNode : Node
*GroupNode : github.com/polarsignals/frostdb/query/logicalplan.Named
func GroupNodeFromThrift(elem *format.SchemaElement, fields FieldList) (*GroupNode, error)
func ListOf(n Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error)
func ListOfWithName(listName string, element Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error)
func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error)
func MustGroup(n Node, err error) *GroupNode
func NewGroupNode(name string, repetition parquet.Repetition, fields FieldList, fieldID int32) (*GroupNode, error)
func NewGroupNodeConverted(name string, repetition parquet.Repetition, fields FieldList, converted ConvertedType, id int32) (n *GroupNode, err error)
func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error)
func (*Schema).Root() *GroupNode
func NewSchema(root *GroupNode) *Schema
func ToThrift(schema *GroupNode) []*format.SchemaElement
IntervalLogicalType is not yet in the thrift spec, but represents
an interval time and needs to be a fixed length byte array of 12 bytes
( IntervalLogicalType) Equals(rhs LogicalType) bool
( IntervalLogicalType) IsApplicable(t parquet.Type, tlen int32) bool
( IntervalLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( IntervalLogicalType) IsNested() bool
( IntervalLogicalType) IsNone() bool
( IntervalLogicalType) IsSerialized() bool
( IntervalLogicalType) IsValid() bool
( IntervalLogicalType) MarshalJSON() ([]byte, error)
( IntervalLogicalType) SortOrder() SortOrder
( IntervalLogicalType) String() string
( IntervalLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
IntervalLogicalType : LogicalType
IntervalLogicalType : github.com/goccy/go-json.Marshaler
IntervalLogicalType : database/sql/driver.Validator
IntervalLogicalType : encoding/json.Marshaler
IntervalLogicalType : expvar.Var
IntervalLogicalType : fmt.Stringer
IntLogicalType represents an integer type of a specific bit width and
is either signed or unsigned.
( IntLogicalType) BitWidth() int8
( IntLogicalType) Equals(rhs LogicalType) bool
( IntLogicalType) IsApplicable(typ parquet.Type, _ int32) bool
( IntLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( IntLogicalType) IsNested() bool
( IntLogicalType) IsNone() bool
( IntLogicalType) IsSerialized() bool
( IntLogicalType) IsSigned() bool
( IntLogicalType) IsValid() bool
( IntLogicalType) MarshalJSON() ([]byte, error)
( IntLogicalType) SortOrder() SortOrder
( IntLogicalType) String() string
( IntLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
IntLogicalType : LogicalType
IntLogicalType : github.com/goccy/go-json.Marshaler
IntLogicalType : database/sql/driver.Validator
IntLogicalType : encoding/json.Marshaler
IntLogicalType : expvar.Var
IntLogicalType : fmt.Stringer
JSONLogicalType represents a byte array column which is to be interpreted
as a JSON string.
( JSONLogicalType) Equals(rhs LogicalType) bool
( JSONLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( JSONLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( JSONLogicalType) IsNested() bool
( JSONLogicalType) IsNone() bool
( JSONLogicalType) IsSerialized() bool
( JSONLogicalType) IsValid() bool
( JSONLogicalType) MarshalJSON() ([]byte, error)
( JSONLogicalType) SortOrder() SortOrder
( JSONLogicalType) String() string
( JSONLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
JSONLogicalType : LogicalType
JSONLogicalType : github.com/goccy/go-json.Marshaler
JSONLogicalType : database/sql/driver.Validator
JSONLogicalType : encoding/json.Marshaler
JSONLogicalType : expvar.Var
JSONLogicalType : fmt.Stringer
ListLogicalType is used for columns which are themselves nested lists
( ListLogicalType) Equals(rhs LogicalType) bool
( ListLogicalType) IsApplicable(parquet.Type, int32) bool
( ListLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool
( ListLogicalType) IsNested() bool
( ListLogicalType) IsNone() bool
( ListLogicalType) IsSerialized() bool
( ListLogicalType) IsValid() bool
( ListLogicalType) MarshalJSON() ([]byte, error)
( ListLogicalType) SortOrder() SortOrder
( ListLogicalType) String() string
( ListLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
ListLogicalType : LogicalType
ListLogicalType : github.com/goccy/go-json.Marshaler
ListLogicalType : database/sql/driver.Validator
ListLogicalType : encoding/json.Marshaler
ListLogicalType : expvar.Var
ListLogicalType : fmt.Stringer
LogicalType is the descriptor that defines the usage of a physical primitive
type in the schema, such as an Interval, Date, etc.
Returns true if the logical types are the same
Returns true if this logical type can be used with the provided physical type
Returns true if the specified ConvertedType is compatible with this
logical type
Returns true if a nested type like List or Map
Returns true if it is NoType
Returns true if this type can be serialized, ie: not Unknown/NoType/Interval
Returns true if not NoLogicalType
Returns the default stat sort order for this logical type
returns a string representation of the Logical Type
Return the equivalent ConvertedType for legacy Parquet systems
BSONLogicalType
DateLogicalType
DecimalLogicalType
EnumLogicalType
Float16LogicalType
IntervalLogicalType
IntLogicalType
JSONLogicalType
ListLogicalType
MapLogicalType
NoLogicalType
NullLogicalType
StringLogicalType
TemporalLogicalType (interface)
TimeLogicalType
TimestampLogicalType
UnknownLogicalType
UUIDLogicalType
VariantLogicalType
LogicalType : database/sql/driver.Validator
LogicalType : expvar.Var
LogicalType : fmt.Stringer
func NewDecimalLogicalType(precision int32, scale int32) LogicalType
func NewIntLogicalType(bitWidth int8, signed bool) LogicalType
func NewListLogicalType() LogicalType
func NewTimeLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func NewTimestampLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func NewTimestampLogicalTypeForce(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func NewTimestampLogicalTypeWithOpts(opts ...TimestampOpt) LogicalType
func (*Column).LogicalType() LogicalType
func ConvertedType.ToLogicalType(convertedDecimal DecimalMetadata) LogicalType
func Node.LogicalType() LogicalType
func github.com/apache/arrow-go/v18/arrow/extensions.(*JSONType).ParquetLogicalType() LogicalType
func github.com/apache/arrow-go/v18/arrow/extensions.(*UUIDType).ParquetLogicalType() LogicalType
func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantType).ParquetLogicalType() LogicalType
func GetLogicalSortOrder(logical LogicalType, primitive format.Type) SortOrder
func NewGroupNodeLogical(name string, repetition parquet.Repetition, fields FieldList, logical LogicalType, id int32) (n *GroupNode, err error)
func NewPrimitiveNodeLogical(name string, repetition parquet.Repetition, logicalType LogicalType, physicalType parquet.Type, typeLen int, id int32) (*PrimitiveNode, error)
func BSONLogicalType.Equals(rhs LogicalType) bool
func DateLogicalType.Equals(rhs LogicalType) bool
func DecimalLogicalType.Equals(rhs LogicalType) bool
func EnumLogicalType.Equals(rhs LogicalType) bool
func Float16LogicalType.Equals(rhs LogicalType) bool
func IntervalLogicalType.Equals(rhs LogicalType) bool
func IntLogicalType.Equals(rhs LogicalType) bool
func JSONLogicalType.Equals(rhs LogicalType) bool
func ListLogicalType.Equals(rhs LogicalType) bool
func LogicalType.Equals(LogicalType) bool
func MapLogicalType.Equals(rhs LogicalType) bool
func NoLogicalType.Equals(rhs LogicalType) bool
func NullLogicalType.Equals(rhs LogicalType) bool
func StringLogicalType.Equals(rhs LogicalType) bool
func TemporalLogicalType.Equals(LogicalType) bool
func TimeLogicalType.Equals(rhs LogicalType) bool
func TimestampLogicalType.Equals(rhs LogicalType) bool
func UnknownLogicalType.Equals(rhs LogicalType) bool
func UUIDLogicalType.Equals(rhs LogicalType) bool
func VariantLogicalType.Equals(rhs LogicalType) bool
MapLogicalType represents a mapped type
( MapLogicalType) Equals(rhs LogicalType) bool
( MapLogicalType) IsApplicable(parquet.Type, int32) bool
( MapLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool
( MapLogicalType) IsNested() bool
( MapLogicalType) IsNone() bool
( MapLogicalType) IsSerialized() bool
( MapLogicalType) IsValid() bool
( MapLogicalType) MarshalJSON() ([]byte, error)
( MapLogicalType) SortOrder() SortOrder
( MapLogicalType) String() string
( MapLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
MapLogicalType : LogicalType
MapLogicalType : github.com/goccy/go-json.Marshaler
MapLogicalType : database/sql/driver.Validator
MapLogicalType : encoding/json.Marshaler
MapLogicalType : expvar.Var
MapLogicalType : fmt.Stringer
Node is the interface for both Group and Primitive Nodes.
A logical schema type has a name, repetition level, and optionally
a logical type (converted type is the deprecated version of the logical
type concept, which is maintained for forward compatibility)
( Node) ConvertedType() ConvertedType
( Node) Equals(Node) bool
( Node) FieldID() int32
( Node) LogicalType() LogicalType
( Node) Name() string
( Node) Parent() Node
( Node) Path() string
( Node) RepetitionType() parquet.Repetition
( Node) SetParent(Node)
( Node) Type() NodeType
( Node) Visit(v Visitor)
*GroupNode
*PrimitiveNode
Node : github.com/polarsignals/frostdb/query/logicalplan.Named
func FromParquet(elems []*format.SchemaElement) (Node, error)
func Must(n Node, err error) Node
func (*Column).SchemaNode() Node
func (*GroupNode).Field(i int) Node
func Node.Parent() Node
func (*Schema).ColumnRoot(i int) Node
func ColumnPathFromNode(n Node) parquet.ColumnPath
func ListOf(n Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error)
func ListOfWithName(listName string, element Node, rep parquet.Repetition, fieldID int32) (*GroupNode, error)
func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error)
func MapOf(name string, key Node, value Node, mapRep parquet.Repetition, fieldID int32) (*GroupNode, error)
func Must(n Node, err error) Node
func MustGroup(n Node, err error) *GroupNode
func MustPrimitive(n Node, err error) *PrimitiveNode
func PrintSchema(n Node, w io.Writer, indentWidth int)
func (*GroupNode).Equals(rhs Node) bool
func (*GroupNode).FieldIndexByField(n Node) int
func Node.Equals(Node) bool
func Node.SetParent(Node)
func (*PrimitiveNode).Equals(rhs Node) bool
func (*Schema).ColumnIndexByNode(n Node) int
func Visitor.VisitPost(Node)
func Visitor.VisitPre(Node) bool
NodeType describes whether the Node is a Primitive or Group node
func Node.Type() NodeType
const Group
const Primitive
( NoLogicalType) Equals(rhs LogicalType) bool
( NoLogicalType) IsApplicable(parquet.Type, int32) bool
( NoLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( NoLogicalType) IsNested() bool
( NoLogicalType) IsNone() bool
( NoLogicalType) IsSerialized() bool
( NoLogicalType) IsValid() bool
( NoLogicalType) MarshalJSON() ([]byte, error)
( NoLogicalType) SortOrder() SortOrder
( NoLogicalType) String() string
( NoLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
NoLogicalType : LogicalType
NoLogicalType : github.com/goccy/go-json.Marshaler
NoLogicalType : database/sql/driver.Validator
NoLogicalType : encoding/json.Marshaler
NoLogicalType : expvar.Var
NoLogicalType : fmt.Stringer
( NullLogicalType) Equals(rhs LogicalType) bool
( NullLogicalType) IsApplicable(parquet.Type, int32) bool
( NullLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( NullLogicalType) IsNested() bool
( NullLogicalType) IsNone() bool
( NullLogicalType) IsSerialized() bool
( NullLogicalType) IsValid() bool
( NullLogicalType) MarshalJSON() ([]byte, error)
( NullLogicalType) SortOrder() SortOrder
( NullLogicalType) String() string
( NullLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
NullLogicalType : LogicalType
NullLogicalType : github.com/goccy/go-json.Marshaler
NullLogicalType : database/sql/driver.Validator
NullLogicalType : encoding/json.Marshaler
NullLogicalType : expvar.Var
NullLogicalType : fmt.Stringer
A PrimitiveNode is a type that is one of the primitive Parquet storage types. In addition to
the other type metadata (name, repetition level, logical type), also has the
physical storage type and their type-specific metadata (byte width, decimal
parameters)
ColumnOrder parquet.ColumnOrder
(*PrimitiveNode) ConvertedType() ConvertedType
DecimalMetadata returns the current metadata for the node. If not a decimal
typed column, the return should have IsSet == false.
Equals returns true if both nodes are primitive nodes with the same physical
and converted/logical types.
(*PrimitiveNode) FieldID() int32
(*PrimitiveNode) LogicalType() LogicalType
(*PrimitiveNode) Name() string
(*PrimitiveNode) Parent() Node
(*PrimitiveNode) Path() string
PhysicalType returns the proper Physical parquet.Type primitive that is used
to store the values in this column.
(*PrimitiveNode) RepetitionType() parquet.Repetition
(*PrimitiveNode) SetParent(p Node)
SetTypeLength will change the type length of the node, has no effect if the
physical type is not FixedLength Byte Array
(*PrimitiveNode) Type() NodeType
TypeLength will be -1 if not a FixedLenByteArray column, otherwise will be the
length of the FixedLen Byte Array
Visit is for implementing a Visitor pattern handler to walk a schema's tree. One
example is the Schema Printer which walks the tree to print out the schema in order.
*PrimitiveNode : Node
*PrimitiveNode : github.com/polarsignals/frostdb/query/logicalplan.Named
func MustPrimitive(n Node, err error) *PrimitiveNode
func NewBooleanNode(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewByteArrayNode(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewFixedLenByteArrayNode(name string, rep parquet.Repetition, length int32, fieldID int32) *PrimitiveNode
func NewFloat32Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewFloat64Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewInt32Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewInt64Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewInt96Node(name string, rep parquet.Repetition, fieldID int32) *PrimitiveNode
func NewPrimitiveNode(name string, repetition parquet.Repetition, typ parquet.Type, fieldID, typeLength int32) (*PrimitiveNode, error)
func NewPrimitiveNodeConverted(name string, repetition parquet.Repetition, typ parquet.Type, converted ConvertedType, typeLen, precision, scale int, id int32) (*PrimitiveNode, error)
func NewPrimitiveNodeLogical(name string, repetition parquet.Repetition, logicalType LogicalType, physicalType parquet.Type, typeLen int, id int32) (*PrimitiveNode, error)
func PrimitiveNodeFromThrift(elem *format.SchemaElement) (*PrimitiveNode, error)
func NewColumn(n *PrimitiveNode, maxDefinitionLvl, maxRepetitionLvl int16) *Column
Schema is the container for the converted Parquet schema with a computed
information from the schema analysis needed for file reading
* Column index to Node
* Max repetition / definition levels for each primitive node
The ColumnDescriptor objects produced by this class can be used to assist in
the reconstruction of fully materialized data structures from the
repetition-definition level encoding of nested data
Column returns the (0-indexed) column of the provided index.
ColumnIndexByName looks up the column by it's full dot separated
node path. If there are multiple columns that match, it returns the first one.
Returns -1 if not found.
ColumnIndexByNode returns the index of the column represented by this node.
Returns -1 if not found.
ColumnRoot returns the root node of a given column if it is under a
nested group node, providing that root group node.
Columns returns an iterator over the leaf columns of the schema
Equals returns true as long as the leaf columns are equal, doesn't take
into account the groups and only checks whether the schemas are compatible
at the physical storage level.
HasRepeatedFields returns true if any node in the schema has a repeated field type.
NumColumns returns the number of leaf nodes that are the actual primitive
columns in this schema.
Root returns the group node that is the root of this schema
(*Schema) String() string
UpdateColumnOrders must get a slice that is the same length as the number of leaf columns
and is used to update the schema metadata Column Orders. len(orders) must equal s.NumColumns()
*Schema : expvar.Var
*Schema : fmt.Stringer
func NewSchema(root *GroupNode) *Schema
func NewSchemaFromStruct(obj interface{}) (sc *Schema, err error)
func NewStructFromSchema(sc *Schema) (t reflect.Type, err error)
func (*Schema).Equals(rhs *Schema) bool
SortOrder mirrors the parquet.thrift sort order type
func DefaultSortOrder(primitive format.Type) SortOrder
func GetLogicalSortOrder(logical LogicalType, primitive format.Type) SortOrder
func GetSortOrder(convert ConvertedType, primitive format.Type) SortOrder
func BSONLogicalType.SortOrder() SortOrder
func (*Column).SortOrder() SortOrder
func DateLogicalType.SortOrder() SortOrder
func DecimalLogicalType.SortOrder() SortOrder
func EnumLogicalType.SortOrder() SortOrder
func Float16LogicalType.SortOrder() SortOrder
func IntervalLogicalType.SortOrder() SortOrder
func IntLogicalType.SortOrder() SortOrder
func JSONLogicalType.SortOrder() SortOrder
func ListLogicalType.SortOrder() SortOrder
func LogicalType.SortOrder() SortOrder
func MapLogicalType.SortOrder() SortOrder
func NoLogicalType.SortOrder() SortOrder
func NullLogicalType.SortOrder() SortOrder
func StringLogicalType.SortOrder() SortOrder
func TemporalLogicalType.SortOrder() SortOrder
func TimeLogicalType.SortOrder() SortOrder
func TimestampLogicalType.SortOrder() SortOrder
func UnknownLogicalType.SortOrder() SortOrder
func UUIDLogicalType.SortOrder() SortOrder
func VariantLogicalType.SortOrder() SortOrder
const SortSIGNED
const SortUNKNOWN
const SortUNSIGNED
StringLogicalType is a UTF8 string, only usable with ByteArray and FixedLenByteArray
( StringLogicalType) Equals(rhs LogicalType) bool
( StringLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( StringLogicalType) IsCompatible(t ConvertedType, dec DecimalMetadata) bool
( StringLogicalType) IsNested() bool
( StringLogicalType) IsNone() bool
( StringLogicalType) IsSerialized() bool
( StringLogicalType) IsValid() bool
( StringLogicalType) MarshalJSON() ([]byte, error)
( StringLogicalType) SortOrder() SortOrder
( StringLogicalType) String() string
( StringLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
StringLogicalType : LogicalType
StringLogicalType : github.com/goccy/go-json.Marshaler
StringLogicalType : database/sql/driver.Validator
StringLogicalType : encoding/json.Marshaler
StringLogicalType : expvar.Var
StringLogicalType : fmt.Stringer
TemporalLogicalType is a smaller interface for Time based logical types
like Time / Timestamp
Returns true if the logical types are the same
( TemporalLogicalType) IsAdjustedToUTC() bool
Returns true if this logical type can be used with the provided physical type
Returns true if the specified ConvertedType is compatible with this
logical type
Returns true if a nested type like List or Map
Returns true if it is NoType
Returns true if this type can be serialized, ie: not Unknown/NoType/Interval
Returns true if not NoLogicalType
Returns the default stat sort order for this logical type
returns a string representation of the Logical Type
( TemporalLogicalType) TimeUnit() TimeUnitType
Return the equivalent ConvertedType for legacy Parquet systems
TimeLogicalType
TimestampLogicalType
TemporalLogicalType : LogicalType
TemporalLogicalType : database/sql/driver.Validator
TemporalLogicalType : expvar.Var
TemporalLogicalType : fmt.Stringer
TimeLogicalType is a time type without a date and must be an
int32 for milliseconds, or an int64 for micro or nano seconds.
( TimeLogicalType) Equals(rhs LogicalType) bool
( TimeLogicalType) IsAdjustedToUTC() bool
( TimeLogicalType) IsApplicable(typ parquet.Type, _ int32) bool
( TimeLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( TimeLogicalType) IsNested() bool
( TimeLogicalType) IsNone() bool
( TimeLogicalType) IsSerialized() bool
( TimeLogicalType) IsValid() bool
( TimeLogicalType) MarshalJSON() ([]byte, error)
( TimeLogicalType) SortOrder() SortOrder
( TimeLogicalType) String() string
( TimeLogicalType) TimeUnit() TimeUnitType
( TimeLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
TimeLogicalType : LogicalType
TimeLogicalType : TemporalLogicalType
TimeLogicalType : github.com/goccy/go-json.Marshaler
TimeLogicalType : database/sql/driver.Validator
TimeLogicalType : encoding/json.Marshaler
TimeLogicalType : expvar.Var
TimeLogicalType : fmt.Stringer
TimestampLogicalType represents an int64 number that can be decoded
into a year, month, day, hour, minute, second, and subsecond
( TimestampLogicalType) Equals(rhs LogicalType) bool
( TimestampLogicalType) IsAdjustedToUTC() bool
( TimestampLogicalType) IsApplicable(t parquet.Type, _ int32) bool
( TimestampLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( TimestampLogicalType) IsFromConvertedType() bool
( TimestampLogicalType) IsNested() bool
( TimestampLogicalType) IsNone() bool
( TimestampLogicalType) IsSerialized() bool
( TimestampLogicalType) IsValid() bool
( TimestampLogicalType) MarshalJSON() ([]byte, error)
( TimestampLogicalType) SortOrder() SortOrder
( TimestampLogicalType) String() string
( TimestampLogicalType) TimeUnit() TimeUnitType
( TimestampLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
TimestampLogicalType : LogicalType
TimestampLogicalType : TemporalLogicalType
TimestampLogicalType : github.com/goccy/go-json.Marshaler
TimestampLogicalType : database/sql/driver.Validator
TimestampLogicalType : encoding/json.Marshaler
TimestampLogicalType : expvar.Var
TimestampLogicalType : fmt.Stringer
TimestampOpt options used with New Timestamp Logical Type
func WithTSForceConverted() TimestampOpt
func WithTSFromConverted() TimestampOpt
func WithTSIsAdjustedToUTC() TimestampOpt
func WithTSTimeUnitType(unit TimeUnitType) TimestampOpt
func NewTimestampLogicalTypeWithOpts(opts ...TimestampOpt) LogicalType
TimeUnitType is an enum for denoting whether a time based logical type
is using milliseconds, microseconds or nanoseconds.
func TemporalLogicalType.TimeUnit() TimeUnitType
func TimeLogicalType.TimeUnit() TimeUnitType
func TimestampLogicalType.TimeUnit() TimeUnitType
func NewTimeLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func NewTimestampLogicalType(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func NewTimestampLogicalTypeForce(isAdjustedToUTC bool, unit TimeUnitType) LogicalType
func WithTSTimeUnitType(unit TimeUnitType) TimestampOpt
const TimeUnitMicros
const TimeUnitMillis
const TimeUnitNanos
const TimeUnitUnknown
UnknownLogicalType is a type that is essentially a placeholder for when
we don't know the type.
( UnknownLogicalType) Equals(rhs LogicalType) bool
( UnknownLogicalType) IsApplicable(parquet.Type, int32) bool
( UnknownLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( UnknownLogicalType) IsNested() bool
( UnknownLogicalType) IsNone() bool
( UnknownLogicalType) IsSerialized() bool
( UnknownLogicalType) IsValid() bool
( UnknownLogicalType) MarshalJSON() ([]byte, error)
( UnknownLogicalType) SortOrder() SortOrder
( UnknownLogicalType) String() string
( UnknownLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
UnknownLogicalType : LogicalType
UnknownLogicalType : github.com/goccy/go-json.Marshaler
UnknownLogicalType : database/sql/driver.Validator
UnknownLogicalType : encoding/json.Marshaler
UnknownLogicalType : expvar.Var
UnknownLogicalType : fmt.Stringer
UUIDLogicalType can only be used with a FixedLength byte array column
that is exactly 16 bytes long
( UUIDLogicalType) Equals(rhs LogicalType) bool
( UUIDLogicalType) IsApplicable(t parquet.Type, tlen int32) bool
( UUIDLogicalType) IsCompatible(c ConvertedType, dec DecimalMetadata) bool
( UUIDLogicalType) IsNested() bool
( UUIDLogicalType) IsNone() bool
( UUIDLogicalType) IsSerialized() bool
( UUIDLogicalType) IsValid() bool
( UUIDLogicalType) MarshalJSON() ([]byte, error)
( UUIDLogicalType) SortOrder() SortOrder
( UUIDLogicalType) String() string
( UUIDLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
UUIDLogicalType : LogicalType
UUIDLogicalType : github.com/goccy/go-json.Marshaler
UUIDLogicalType : database/sql/driver.Validator
UUIDLogicalType : encoding/json.Marshaler
UUIDLogicalType : expvar.Var
UUIDLogicalType : fmt.Stringer
( VariantLogicalType) Equals(rhs LogicalType) bool
( VariantLogicalType) IsApplicable(parquet.Type, int32) bool
( VariantLogicalType) IsCompatible(ct ConvertedType, _ DecimalMetadata) bool
( VariantLogicalType) IsNested() bool
( VariantLogicalType) IsNone() bool
( VariantLogicalType) IsSerialized() bool
( VariantLogicalType) IsValid() bool
( VariantLogicalType) MarshalJSON() ([]byte, error)
( VariantLogicalType) SortOrder() SortOrder
( VariantLogicalType) String() string
( VariantLogicalType) ToConvertedType() (ConvertedType, DecimalMetadata)
VariantLogicalType : LogicalType
VariantLogicalType : github.com/goccy/go-json.Marshaler
VariantLogicalType : database/sql/driver.Validator
VariantLogicalType : encoding/json.Marshaler
VariantLogicalType : expvar.Var
VariantLogicalType : fmt.Stringer
Visitor is an interface for creating functionality to walk the schema tree.
A visitor can be passed to the Visit function of a Node in order to walk
the tree. VisitPre is called the first time a node is encountered. If
it is a group node, the return is checked and if it is false, the children
will be skipped.
VisitPost is called after visiting any children
( Visitor) VisitPost(Node)
( Visitor) VisitPre(Node) bool
func (*GroupNode).Visit(v Visitor)
func Node.Visit(v Visitor)
func (*PrimitiveNode).Visit(v Visitor)
Package-Level Functions (total 44)
ColumnPathFromNode walks the parents of the given node to construct it's
column path
DefaultSortOrder returns the default stat sort order for the given physical type
FromParquet converts a slice of thrift Schema Elements to the correct node type
GetLogicalSortOrder returns the default sort order for this logical type
or falls back to the default sort order for the physical type if not valid
GetSortOrder defaults to the sort order based on the physical type if convert
is ConvertedTypes.None, otherwise determines the sort order by the converted type.
func GroupNodeFromThrift(elem *format.SchemaElement, fields FieldList) (*GroupNode, error)
ListOf is a convenience helper function to create a properly structured
list structure according to the Parquet Spec.
<list-repetition> group <name> (LIST) {
repeated group list {
<element-repetition> <element-type> element;
}
}
<list-repetition> can only be optional or required.
<element-repetition> can only be optional or required.
ListOf is a convenience helper function to create a properly structured
list structure according to the Parquet Spec.
<list-repetition> group <name> (LIST) {
repeated group list {
<element-repetition> <element-type> element;
}
}
<list-repetition> can only be optional or required.
<element-repetition> can only be optional or required.
MapOf is a convenience helper function to create a properly structured
parquet map node setup according to the Parquet Spec.
<map-repetition> group <name> (MAP) {
repeated group key_value {
required <key-type> key;
<value-repetition> <value-type> value;
}
}
key node will be renamed to "key", value node if not nil will be renamed to "value"
<map-repetition> must be only optional or required. panics if repeated is passed.
the key node *must* be required repetition. panics if optional or repeated
value node can be nil (omitted) or have a repetition of required or optional *only*.
Must is a convenience function for the NewNode functions that return a Node
and an error, panic'ing if err != nil or returning the node
MustGroup is like Must, except it casts the node to a *GroupNode, which will panic
if it is a primitive node.
MustPrimitive is like Must except it casts the node to *PrimitiveNode which will panic
if it is a group node.
NewBooleanNode is a convenience factory for constructing an Boolean Primitive Node
NewByteArrayNode is a convenience factory for constructing an Byte Array Primitive Node
NewColumn returns a new column object for the given node with the provided
maximum definition and repetition levels.
NewDecimalLogicalType returns a Decimal logical type with the given
precision and scale.
Panics if precision < 1 or scale is not in the range (0, precision)
NewFixedLenByteArrayNode is a convenience factory for constructing an Fixed Length
Byte Array Primitive Node of the given length
NewFloat32Node is a convenience factory for constructing an Float Primitive Node
NewFloat64Node is a convenience factory for constructing an Double Primitive Node
NewGroupNode constructs a new group node with the provided fields,
but with converted type None and No Logical Type
NewGroupNodeConverted constructs a group node with the provided fields and converted type,
determining the logical type from that converted type.
NewGroupNodeLogical constructs a group node with the provided fields and logical type,
determining the converted type from the provided logical type.
NewInt32Node is a convenience factory for constructing an Int32 Primitive Node
NewInt64Node is a convenience factory for constructing an Int64 Primitive Node
NewInt96Node is a convenience factory for constructing an Int96 Primitive Node
NewIntLogicalType creates an integer logical type of the desired bitwidth
and whether it is signed or not.
Bit width must be exactly 8, 16, 32 or 64 for an integer logical type
func NewListLogicalType() LogicalType
NewPrimitiveNode constructs a primitive node with the ConvertedType of None and no logical type.
Use NewPrimitiveNodeLogical and NewPrimitiveNodeConverted to specify the logical or converted type.
NewPrimitiveNodeConverted constructs a primitive node from the given physical type and converted type,
determining the logical type from the converted type.
NewPrimitiveNodeLogical constructs a Primitive node using the provided logical type for a given
physical type and typelength.
NewSchema constructs a new Schema object from a root group node.
Any fields with a field-id of -1 will be given an appropriate field number based on their order.
NewSchemaFromStruct generates a schema from an object type via reflection of
the type and reading struct tags for "parquet".
# Rules
Everything defaults to Required repetition, unless otherwise specified.
Pointer types become Optional repetition.
Arrays and Slices become logical List types unless using the tag `repetition=repeated`.
A length specified byte field (like [5]byte) becomes a fixed_len_byte_array of that length
unless otherwise specified by tags.
string and []byte both become ByteArray unless otherwise specified.
Integer types will default to having a logical type of the appropriate bit width
and signedness rather than having no logical type, ie: an int8 will become an int32
node with logical type Int(bitWidth=8, signed=true).
Structs will become group nodes with the fields of the struct as the fields of the group,
recursively creating the nodes.
maps will become appropriate Map structures in the schema of the defined key and values.
# Available Tags
name: by default the node will have the same name as the field, this tag let's you specify a name
type: Specify the physical type instead of using the field type
length: specify the type length of the node, only relevant for fixed_len_byte_array
scale: specify the scale for a decimal field
precision: specify the precision for a decimal field
fieldid: specify the field ID for that node, defaults to -1 which means it is not set in the parquet file.
repetition: specify the repetition as something other than what is determined by the type
converted: specify the Converted Type of the field
logical: specify the logical type of the field, if using decimal then the scale and precision
will be determined by the precision and scale fields, or by the logical.precision / logical.scale fields
with the logical. prefixed versions taking precedence. For Time or Timestamp logical types,
use logical.unit=<millis|micros|nanos> and logical.isadjustedutc=<true|false> to set those. Unit is required
isadjustedutc defaults to true. For Integer logical type, use logical.bitwidth and logical.signed to specify
those values, with bitwidth being required, and signed defaulting to true.
All tags other than name can use a prefix of "key<tagname>=<value>" to refer to the type of the key for a map
and "value<tagname>=<value>" to refer to the value type of a map or the element of a list (such as the type of a slice)
NewStructFromSchema generates a struct type as a reflect.Type from the schema
by using the appropriate physical types and making things either pointers or slices
based on whether they are repeated/optional/required. It does not use the logical
or converted types to change the physical storage so that it is more efficient to use
the resulting type for reading without having to do conversions.
It will use maps for map types and slices for list types, but otherwise ignores the
converted and logical types of the nodes. Group nodes that are not List or Map will
be nested structs.
NewTimeLogicalType returns a time type of the given unit.
NewTimestampLogicalType returns a logical timestamp type with "forceConverted"
set to false
NewTimestampLogicalTypeForce returns a timestamp logical type with
"forceConverted" set to true
NewTimestampLogicalTypeWithOpts creates a new TimestampLogicalType with the provided options.
TimestampType Unit defaults to milliseconds (TimeUnitMillis)
func PrimitiveNodeFromThrift(elem *format.SchemaElement) (*PrimitiveNode, error)
PrintSchema writes a string representation of the tree to w using the indent
width provided.
ToThrift converts a GroupNode to a slice of SchemaElements which is used
for thrift serialization.
WithTSForceConverted enable force converted mode
WithTSFromConverted enable the timestamp logical type to be
constructed from a converted type.
WithTSIsAdjustedToUTC sets the IsAdjustedToUTC field of the timestamp type.
WithTSTimeUnitType sets the time unit for the timestamp type
Package-Level Variables (only one)
ConvertedTypes is a struct containing the constants for the types
to make it easy to reference them while making it clear what they are
Package-Level Constants (total 9)
the available constants for NodeType
the available constants for NodeType
Constants for the Stat sort order definitions
Constants for the Stat sort order definitions
Constants for the Stat sort order definitions
Constants for the TimeUnitType
Constants for the TimeUnitType
Constants for the TimeUnitType
Constants for the TimeUnitType
![]() |
The pages are generated with Golds v0.8.2. (GOOS=linux GOARCH=amd64) Golds is a Go 101 project developed by Tapir Liu. PR and bug reports are welcome and can be submitted to the issue list. Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds. |