ArrayData is the underlying memory and metadata of an Arrow array, corresponding
to the same-named object in the C++ implementation.
The Array interface and subsequent typed objects provide strongly typed
accessors which support marshalling and other patterns to the data.
This interface allows direct access to the underlying raw byte buffers
which allows for manipulating the internal data and casting. For example,
one could cast the raw bytes from int64 to float64 like so:
arrdata := GetMyInt64Data().Data()
newdata := array.NewData(arrow.PrimitiveTypes.Float64, arrdata.Len(),
arrdata.Buffers(), nil, arrdata.NullN(), arrdata.Offset())
defer newdata.Release()
float64arr := array.NewFloat64Data(newdata)
defer float64arr.Release()
This is also useful in an analytics setting where memory may be reused. For
example, if we had a group of operations all returning float64 such as:
Log(Sqrt(Expr(arr)))
The low-level implementations could have signatures such as:
func Log(values arrow.ArrayData) arrow.ArrayData
Another example would be a function that consumes one or more memory buffers
in an input array and replaces them with newly-allocated data, changing the
output data type as well. Buffers returns the slice of raw data buffers for this data instance. Their
meaning depends on the context of the data type. Children returns the slice of children data instances, only relevant for
nested data types. For instance, List data will have a single child containing
elements of all the rows and Struct data will contain numfields children which
are the arrays for each field of the struct. DataType returns the current datatype stored in the object. Dictionary returns the ArrayData object for the dictionary if this is a
dictionary array, otherwise it will be nil. Len returns the length of this data instance NullN returns the number of nulls for this data instance. Offset returns the offset into the raw buffers where this data begins Release decreases the reference count by 1, it is safe to call
in multiple goroutines simultaneously. Data is removed when reference
count is 0. Reset allows reusing this ArrayData object by replacing the data in this ArrayData
object without changing the reference count. Retain increases the reference count by 1, it is safe to call
in multiple goroutines simultaneously. SizeInBytes returns the size of the ArrayData buffers and any children and/or dictionary in bytes.
*github.com/apache/arrow-go/v18/arrow/array.Data
ArrayData : github.com/apache/arrow-go/v18/arrow/scalar.Releasable
func Array.Data() ArrayData
func ArrayData.Children() []ArrayData
func ArrayData.Dictionary() ArrayData
func TypedArray.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.NewSliceData(data ArrayData, i, j int64) ArrayData
func github.com/apache/arrow-go/v18/arrow/array.TransposeDictIndices(mem memory.Allocator, data ArrayData, inType, outType DataType, dict ArrayData, transposeMap []int32) (ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/array.BinaryLike.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.(*Data).Children() []ArrayData
func github.com/apache/arrow-go/v18/arrow/array.(*Data).Dictionary() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.ExtensionArray.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.ListLike.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.StringLike.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.Union.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.VarLenListLike.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/array.ViewLike.Data() ArrayData
func github.com/apache/arrow-go/v18/arrow/compute/exec.(*ArraySpan).MakeData() ArrayData
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.GetTakeIndices(mem memory.Allocator, filter *exec.ArraySpan, nullSelect kernels.NullSelectionBehavior) (ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.HashState.GetDictionary() (ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.(*Memo).Dict(id int64, mem memory.Allocator) (ArrayData, error)
func github.com/polarsignals/frostdb/pqarrow/arrowutils.VirtualNullArray.Data() ArrayData
func GetOffsets[T](data ArrayData, i int) []T
func GetValues[T](data ArrayData, i int) []T
func ArrayData.Reset(newtype DataType, newlength int, newbuffers []*memory.Buffer, newchildren []ArrayData, newnulls int, newoffset int)
func github.com/apache/arrow-go/v18/arrow/array.Hash(h *maphash.Hash, data ArrayData)
func github.com/apache/arrow-go/v18/arrow/array.MakeFromData(data ArrayData) Array
func github.com/apache/arrow-go/v18/arrow/array.NewBinaryData(data ArrayData) *array.Binary
func github.com/apache/arrow-go/v18/arrow/array.NewBinaryViewData(data ArrayData) *array.BinaryView
func github.com/apache/arrow-go/v18/arrow/array.NewBooleanData(data ArrayData) *array.Boolean
func github.com/apache/arrow-go/v18/arrow/array.NewData(dtype DataType, length int, buffers []*memory.Buffer, childData []ArrayData, nulls, offset int) *array.Data
func github.com/apache/arrow-go/v18/arrow/array.NewDate32Data(data ArrayData) *array.Date32
func github.com/apache/arrow-go/v18/arrow/array.NewDate64Data(data ArrayData) *array.Date64
func github.com/apache/arrow-go/v18/arrow/array.NewDayTimeIntervalData(data ArrayData) *array.DayTimeInterval
func github.com/apache/arrow-go/v18/arrow/array.NewDecimal128Data(data ArrayData) *array.Decimal128
func github.com/apache/arrow-go/v18/arrow/array.NewDecimal256Data(data ArrayData) *array.Decimal256
func github.com/apache/arrow-go/v18/arrow/array.NewDecimal32Data(data ArrayData) *array.Decimal32
func github.com/apache/arrow-go/v18/arrow/array.NewDecimal64Data(data ArrayData) *array.Decimal64
func github.com/apache/arrow-go/v18/arrow/array.NewDenseUnionData(data ArrayData) *array.DenseUnion
func github.com/apache/arrow-go/v18/arrow/array.NewDictionaryData(data ArrayData) *array.Dictionary
func github.com/apache/arrow-go/v18/arrow/array.NewDurationData(data ArrayData) *array.Duration
func github.com/apache/arrow-go/v18/arrow/array.NewExtensionData(data ArrayData) array.ExtensionArray
func github.com/apache/arrow-go/v18/arrow/array.NewFixedSizeBinaryData(data ArrayData) *array.FixedSizeBinary
func github.com/apache/arrow-go/v18/arrow/array.NewFixedSizeListData(data ArrayData) *array.FixedSizeList
func github.com/apache/arrow-go/v18/arrow/array.NewFloat16Data(data ArrayData) *array.Float16
func github.com/apache/arrow-go/v18/arrow/array.NewFloat32Data(data ArrayData) *array.Float32
func github.com/apache/arrow-go/v18/arrow/array.NewFloat64Data(data ArrayData) *array.Float64
func github.com/apache/arrow-go/v18/arrow/array.NewInt16Data(data ArrayData) *array.Int16
func github.com/apache/arrow-go/v18/arrow/array.NewInt32Data(data ArrayData) *array.Int32
func github.com/apache/arrow-go/v18/arrow/array.NewInt64Data(data ArrayData) *array.Int64
func github.com/apache/arrow-go/v18/arrow/array.NewInt8Data(data ArrayData) *array.Int8
func github.com/apache/arrow-go/v18/arrow/array.NewIntervalData(data ArrayData) Array
func github.com/apache/arrow-go/v18/arrow/array.NewLargeBinaryData(data ArrayData) *array.LargeBinary
func github.com/apache/arrow-go/v18/arrow/array.NewLargeListData(data ArrayData) *array.LargeList
func github.com/apache/arrow-go/v18/arrow/array.NewLargeListViewData(data ArrayData) *array.LargeListView
func github.com/apache/arrow-go/v18/arrow/array.NewLargeStringData(data ArrayData) *array.LargeString
func github.com/apache/arrow-go/v18/arrow/array.NewListData(data ArrayData) *array.List
func github.com/apache/arrow-go/v18/arrow/array.NewListViewData(data ArrayData) *array.ListView
func github.com/apache/arrow-go/v18/arrow/array.NewMapData(data ArrayData) *array.Map
func github.com/apache/arrow-go/v18/arrow/array.NewMonthDayNanoIntervalData(data ArrayData) *array.MonthDayNanoInterval
func github.com/apache/arrow-go/v18/arrow/array.NewMonthIntervalData(data ArrayData) *array.MonthInterval
func github.com/apache/arrow-go/v18/arrow/array.NewNullData(data ArrayData) *array.Null
func github.com/apache/arrow-go/v18/arrow/array.NewRunEndEncodedData(data ArrayData) *array.RunEndEncoded
func github.com/apache/arrow-go/v18/arrow/array.NewSliceData(data ArrayData, i, j int64) ArrayData
func github.com/apache/arrow-go/v18/arrow/array.NewSparseUnionData(data ArrayData) *array.SparseUnion
func github.com/apache/arrow-go/v18/arrow/array.NewStringData(data ArrayData) *array.String
func github.com/apache/arrow-go/v18/arrow/array.NewStringViewData(data ArrayData) *array.StringView
func github.com/apache/arrow-go/v18/arrow/array.NewStructData(data ArrayData) *array.Struct
func github.com/apache/arrow-go/v18/arrow/array.NewTime32Data(data ArrayData) *array.Time32
func github.com/apache/arrow-go/v18/arrow/array.NewTime64Data(data ArrayData) *array.Time64
func github.com/apache/arrow-go/v18/arrow/array.NewTimestampData(data ArrayData) *array.Timestamp
func github.com/apache/arrow-go/v18/arrow/array.NewUint16Data(data ArrayData) *array.Uint16
func github.com/apache/arrow-go/v18/arrow/array.NewUint32Data(data ArrayData) *array.Uint32
func github.com/apache/arrow-go/v18/arrow/array.NewUint64Data(data ArrayData) *array.Uint64
func github.com/apache/arrow-go/v18/arrow/array.NewUint8Data(data ArrayData) *array.Uint8
func github.com/apache/arrow-go/v18/arrow/array.TransposeDictIndices(mem memory.Allocator, data ArrayData, inType, outType DataType, dict ArrayData, transposeMap []int32) (ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/array.TransposeDictIndices(mem memory.Allocator, data ArrayData, inType, outType DataType, dict ArrayData, transposeMap []int32) (ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/array.(*Data).Reset(dtype DataType, length int, buffers []*memory.Buffer, childData []ArrayData, nulls, offset int)
func github.com/apache/arrow-go/v18/arrow/array.(*Data).SetDictionary(dict ArrayData)
func github.com/apache/arrow-go/v18/arrow/array.(*LargeString).Reset(data ArrayData)
func github.com/apache/arrow-go/v18/arrow/array.(*String).Reset(data ArrayData)
func github.com/apache/arrow-go/v18/arrow/array.(*StringView).Reset(data ArrayData)
func github.com/apache/arrow-go/v18/arrow/compute/exec.(*ArraySpan).SetMembers(data ArrayData)
func github.com/apache/arrow-go/v18/arrow/compute/exec.(*ArraySpan).TakeOwnership(data ArrayData)
func github.com/apache/arrow-go/v18/arrow/encoded.FindPhysicalIndex(arr ArrayData, logicalIdx int) int
func github.com/apache/arrow-go/v18/arrow/encoded.FindPhysicalOffset(arr ArrayData) int
func github.com/apache/arrow-go/v18/arrow/encoded.GetPhysicalLength(arr ArrayData) int
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.ResolveDictionaries(memo *dictutils.Memo, cols []ArrayData, parentPos dictutils.FieldPos, mem memory.Allocator) error
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.ResolveFieldDict(memo *dictutils.Memo, data ArrayData, pos dictutils.FieldPos, mem memory.Allocator) error
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.(*Memo).Add(id int64, v ArrayData)
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.(*Memo).AddDelta(id int64, v ArrayData)
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.(*Memo).AddOrReplace(id int64, v ArrayData) bool
func github.com/apache/arrow-go/v18/arrow/internal/dictutils.Memo.HasDict(v ArrayData) bool
func github.com/apache/arrow-go/v18/arrow/scalar.NewLargeListScalarData(val ArrayData) *scalar.LargeList
func github.com/apache/arrow-go/v18/arrow/scalar.NewListScalarData(val ArrayData) *scalar.List
BitWidth returns the number of bits required to store a single element of this data type in memory.( BooleanType) Bytes() int(*BooleanType) Fingerprint() string(*BooleanType) ID() Type( BooleanType) Layout() DataTypeLayout(*BooleanType) Name() string(*BooleanType) String() string
*BooleanType : DataType
*BooleanType : FixedWidthDataType
*BooleanType : github.com/polarsignals/frostdb/query/logicalplan.Named
*BooleanType : expvar.Var
*BooleanType : fmt.Stringer
Chunked manages a collection of primitives arrays as one logical large array.(*Chunked) Chunk(i int) Array(*Chunked) Chunks() []Array(*Chunked) DataType() DataType(*Chunked) Len() int(*Chunked) NullN() int Release decreases the reference count by 1.
When the reference count goes to zero, the memory is freed.
Release may be called simultaneously from multiple goroutines. Retain increases the reference count by 1.
Retain may be called simultaneously from multiple goroutines.
*Chunked : github.com/apache/arrow-go/v18/arrow/scalar.Releasable
func NewChunked(dtype DataType, chunks []Array) *Chunked
func (*Column).Data() *Chunked
func github.com/apache/arrow-go/v18/arrow/array.ChunkedFromJSON(mem memory.Allocator, dt DataType, chunkStrs []string, opts ...array.FromJSONOption) (*Chunked, error)
func github.com/apache/arrow-go/v18/arrow/array.NewChunkedSlice(a *Chunked, i, j int64) *Chunked
func github.com/apache/arrow-go/v18/arrow/array.UnifyChunkedDicts(alloc memory.Allocator, chnkd *Chunked) (*Chunked, error)
func NewColumn(field Field, chunks *Chunked) *Column
func github.com/apache/arrow-go/v18/arrow/array.ChunkedApproxEqual(left, right *Chunked, opts ...array.EqualOption) bool
func github.com/apache/arrow-go/v18/arrow/array.ChunkedEqual(left, right *Chunked) bool
func github.com/apache/arrow-go/v18/arrow/array.NewChunkedSlice(a *Chunked, i, j int64) *Chunked
func github.com/apache/arrow-go/v18/arrow/array.UnifyChunkedDicts(alloc memory.Allocator, chnkd *Chunked) (*Chunked, error)
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.ChunkedPrimitiveTake(ctx *exec.KernelCtx, batch []*Chunked, out *exec.ExecResult) ([]*exec.ExecResult, error)
Column is an immutable column data structure consisting of
a field (type metadata) and a chunked data array.
To get strongly typed data from a Column, you need to iterate the
chunks and type assert each individual Array. For example:
switch column.DataType().ID() {
case arrow.INT32:
for _, c := range column.Data().Chunks() {
arr := c.(*array.Int32)
// do something with arr
}
case arrow.INT64:
for _, c := range column.Data().Chunks() {
arr := c.(*array.Int64)
// do something with arr
}
case ...
}(*Column) Data() *Chunked(*Column) DataType() DataType(*Column) Field() Field(*Column) Len() int(*Column) Name() string(*Column) NullN() int Release decreases the reference count by 1.
When the reference count goes to zero, the memory is freed.
Release may be called simultaneously from multiple goroutines. Retain increases the reference count by 1.
Retain may be called simultaneously from multiple goroutines.
*Column : github.com/apache/arrow-go/v18/arrow/scalar.Releasable
*Column : github.com/polarsignals/frostdb/query/logicalplan.Named
func NewColumn(field Field, chunks *Chunked) *Column
func NewColumnFromArr(field Field, arr Array) Column
func Table.Column(i int) *Column
func github.com/apache/arrow-go/v18/arrow/array.NewColumnSlice(col *Column, i, j int64) *Column
func Table.AddColumn(pos int, f Field, c Column) (Table, error)
func github.com/apache/arrow-go/v18/arrow/array.NewColumnSlice(col *Column, i, j int64) *Column
func github.com/apache/arrow-go/v18/arrow/array.NewTable(schema *Schema, cols []Column, rows int64) Table
DayTimeIntervalType is encoded as a pair of 32-bit signed integer,
representing a number of days and milliseconds (fraction of day). BitWidth returns the number of bits required to store a single element of this data type in memory.( DayTimeIntervalType) Bytes() int(*DayTimeIntervalType) Fingerprint() string(*DayTimeIntervalType) ID() Type( DayTimeIntervalType) Layout() DataTypeLayout(*DayTimeIntervalType) Name() string(*DayTimeIntervalType) String() string
*DayTimeIntervalType : DataType
*DayTimeIntervalType : FixedWidthDataType
*DayTimeIntervalType : github.com/polarsignals/frostdb/query/logicalplan.Named
*DayTimeIntervalType : expvar.Var
*DayTimeIntervalType : fmt.Stringer
DenseUnionType is the concrete type for dense union data.
A dense union is a nested type where each logical value is taken from a
single child, at a specific offset. A buffer of 8-bit type ids (typed
as UnionTypeCode) indicates which child a given logical value is to be
taken from and a buffer of 32-bit offsets indicating which physical position
in the given child array has the logical value for that index.
Unlike a sparse union, a dense union allows encoding only the child values
which are actually referred to by the union array. This is counterbalanced
by the additional footprint of the offsets buffer, and the additional
indirection cost when looking up values.
Unlike most other types, unions don't have a top-level validity bitmap(*DenseUnionType) ChildIDs() []int Fields method provides a copy of union type fields
(so it can be safely mutated and will not result in updating the union type).(*DenseUnionType) Fingerprint() string( DenseUnionType) ID() Type( DenseUnionType) Layout() DataTypeLayout(*DenseUnionType) MaxTypeCode() (max UnionTypeCode)( DenseUnionType) Mode() UnionMode( DenseUnionType) Name() string(*DenseUnionType) NumFields() int( DenseUnionType) OffsetTypeTraits() OffsetTraits(*DenseUnionType) String() string(*DenseUnionType) TypeCodes() []UnionTypeCode
*DenseUnionType : DataType
*DenseUnionType : NestedType
*DenseUnionType : OffsetsDataType
*DenseUnionType : UnionType
DenseUnionType : github.com/polarsignals/frostdb/query/logicalplan.Named
*DenseUnionType : expvar.Var
*DenseUnionType : fmt.Stringer
func DenseUnionFromArrays(children []Array, fields []string, codes []UnionTypeCode) *DenseUnionType
func DenseUnionOf(fields []Field, typeCodes []UnionTypeCode) *DenseUnionType
func github.com/apache/arrow-go/v18/arrow/array.NewDenseUnion(dt *DenseUnionType, length int, children []Array, typeIDs, valueOffsets *memory.Buffer, offset int) *array.DenseUnion
func github.com/apache/arrow-go/v18/arrow/array.NewDenseUnionBuilder(mem memory.Allocator, typ *DenseUnionType) *array.DenseUnionBuilder
func github.com/apache/arrow-go/v18/arrow/array.NewDenseUnionBuilderWithBuilders(mem memory.Allocator, typ *DenseUnionType, children []array.Builder) *array.DenseUnionBuilder
func github.com/apache/arrow-go/v18/arrow/scalar.NewDenseUnionScalar(v scalar.Scalar, code UnionTypeCode, dt *DenseUnionType) *scalar.DenseUnion
ExtensionBase is the base struct for user-defined Extension Types which must be
embedded in any user-defined types like so:
type UserDefinedType struct {
arrow.ExtensionBase
// any other data
} Storage is the underlying storage type(*ExtensionBase) Fields() []Field(*ExtensionBase) Fingerprint() string ID always returns arrow.EXTENSION and should not be overridden(*ExtensionBase) Layout() DataTypeLayout Name should always return "extension" and should not be overridden(*ExtensionBase) NumFields() int StorageType returns the underlying storage type and exists so that functions
written against the ExtensionType interface can access the storage type. String by default will return "extension_type<storage=storage_type>" by can be overridden
to customize what is printed out when printing this extension type.
*ExtensionBase : DataType
*ExtensionBase : NestedType
*ExtensionBase : github.com/polarsignals/frostdb/query/logicalplan.Named
*ExtensionBase : expvar.Var
*ExtensionBase : fmt.Stringer
ExtensionType is an interface for handling user-defined types. They must be
DataTypes and must embed arrow.ExtensionBase in them in order to work properly
ensuring that they always have the expected base behavior.
The arrow.ExtensionBase that needs to be embedded implements the DataType interface
leaving the remaining functions having to be implemented by the actual user-defined
type in order to be handled properly. ArrayType should return the reflect.TypeOf(ExtensionArrayType{}) where the
ExtensionArrayType is a type that implements the array.ExtensionArray interface.
Such a type must also embed the array.ExtensionArrayBase in it. This will be used
when creating arrays of this ExtensionType by using reflect.New Deserialize is called when reading in extension arrays and types via the IPC format
in order to construct an instance of the appropriate extension type. The passed in data
is pulled from the ARROW:extension:metadata key and may be nil or an empty slice.
If the storage type is incorrect or something else is invalid with the data this should
return nil and an appropriate error. ExtensionEquals is used to tell whether two ExtensionType instances are equal types. ExtensionName is what will be used when registering / unregistering this extension
type. Multiple user-defined types can be defined with a parameterized ExtensionType
as long as the parameter is used in the ExtensionName to distinguish the instances
in the global Extension Type registry.
The return from this is also what will be placed in the metadata for IPC communication
under the key ARROW:extension:name( ExtensionType) Fingerprint() string( ExtensionType) ID() Type( ExtensionType) Layout() DataTypeLayout Name is name of the data type. Serialize should produce any extra metadata necessary for initializing an instance of
this user-defined type. Not all user-defined types require this and it is valid to return
nil from this function or an empty slice. This is used for the IPC format and will be
added to metadata for IPC communication under the key ARROW:extension:metadata
This should be implemented such that it is valid to be called by multiple goroutines
concurrently. StorageType returns the underlying storage type which is used by this extension
type. It is already implemented by the ExtensionBase struct and thus does not need
to be re-implemented by a user-defined type.( ExtensionType) String() string
*github.com/apache/arrow-go/v18/arrow/extensions.Bool8Type
*github.com/apache/arrow-go/v18/arrow/extensions.JSONType
*github.com/apache/arrow-go/v18/arrow/extensions.OpaqueType
*github.com/apache/arrow-go/v18/arrow/extensions.UUIDType
*github.com/apache/arrow-go/v18/arrow/extensions.VariantType
ExtensionType : DataType
ExtensionType : github.com/polarsignals/frostdb/query/logicalplan.Named
ExtensionType : expvar.Var
ExtensionType : fmt.Stringer
func GetExtensionType(typName string) ExtensionType
func ExtensionType.Deserialize(storageType DataType, data string) (ExtensionType, error)
func github.com/apache/arrow-go/v18/arrow/array.ExtensionArray.ExtensionType() ExtensionType
func github.com/apache/arrow-go/v18/arrow/array.(*ExtensionArrayBase).ExtensionType() ExtensionType
func github.com/apache/arrow-go/v18/arrow/extensions.(*Bool8Type).Deserialize(storageType DataType, data string) (ExtensionType, error)
func github.com/apache/arrow-go/v18/arrow/extensions.(*JSONType).Deserialize(storageType DataType, data string) (ExtensionType, error)
func github.com/apache/arrow-go/v18/arrow/extensions.(*OpaqueType).Deserialize(storageType DataType, data string) (ExtensionType, error)
func github.com/apache/arrow-go/v18/arrow/extensions.(*UUIDType).Deserialize(storageType DataType, data string) (ExtensionType, error)
func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantType).Deserialize(storageType DataType, _ string) (ExtensionType, error)
func RegisterExtensionType(typ ExtensionType) error
func ExtensionType.ExtensionEquals(ExtensionType) bool
func github.com/apache/arrow-go/v18/arrow/array.NewExtensionArrayWithStorage(dt ExtensionType, storage Array) Array
func github.com/apache/arrow-go/v18/arrow/array.NewExtensionBuilder(mem memory.Allocator, dt ExtensionType) *array.ExtensionBuilder
func github.com/apache/arrow-go/v18/arrow/extensions.(*Bool8Type).ExtensionEquals(other ExtensionType) bool
func github.com/apache/arrow-go/v18/arrow/extensions.(*JSONType).ExtensionEquals(other ExtensionType) bool
func github.com/apache/arrow-go/v18/arrow/extensions.(*OpaqueType).ExtensionEquals(other ExtensionType) bool
func github.com/apache/arrow-go/v18/arrow/extensions.(*UUIDType).ExtensionEquals(other ExtensionType) bool
func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantType).ExtensionEquals(other ExtensionType) bool
FixedWidthType is a type constraint for raw values in Arrow that
can be represented as FixedWidth byte slices. Specifically this is for
using Go generics to easily re-type a byte slice to a properly-typed
slice. Booleans are excluded here since they are represented by Arrow
as a bitmap and thus the buffer can't be just reinterpreted as a []bool
Float16Type represents a floating point value encoded with a 16-bit precision. BitWidth returns the number of bits required to store a single element of this data type in memory.( Float16Type) Bytes() int(*Float16Type) Fingerprint() string(*Float16Type) ID() Type( Float16Type) Layout() DataTypeLayout(*Float16Type) Name() string(*Float16Type) String() string
*Float16Type : DataType
*Float16Type : FixedWidthDataType
*Float16Type : github.com/polarsignals/frostdb/query/logicalplan.Named
*Float16Type : expvar.Var
*Float16Type : fmt.Stringer
IntType is a type constraint for raw values represented as signed
integer types by We aren't just using constraints.Signed
because we don't want to include the raw `int` type here whose size
changes based on the architecture (int32 on 32-bit architectures and
int64 on 64-bit architectures).
This will also cover types like MonthInterval or the time types
as their underlying types are int32 and int64 which will get covered
by using the ~
MonthDayNanoIntervalType is encoded as two signed 32-bit integers representing
a number of months and a number of days, followed by a 64-bit integer representing
the number of nanoseconds since midnight for fractions of a day. BitWidth returns the number of bits required to store a single element of this data type in memory.(*MonthDayNanoIntervalType) Bytes() int(*MonthDayNanoIntervalType) Fingerprint() string(*MonthDayNanoIntervalType) ID() Type( MonthDayNanoIntervalType) Layout() DataTypeLayout(*MonthDayNanoIntervalType) Name() string(*MonthDayNanoIntervalType) String() string
*MonthDayNanoIntervalType : DataType
*MonthDayNanoIntervalType : FixedWidthDataType
*MonthDayNanoIntervalType : github.com/polarsignals/frostdb/query/logicalplan.Named
*MonthDayNanoIntervalType : expvar.Var
*MonthDayNanoIntervalType : fmt.Stringer
MonthIntervalType is encoded as a 32-bit signed integer,
representing a number of months. BitWidth returns the number of bits required to store a single element of this data type in memory.( MonthIntervalType) Bytes() int(*MonthIntervalType) Fingerprint() string(*MonthIntervalType) ID() Type( MonthIntervalType) Layout() DataTypeLayout(*MonthIntervalType) Name() string(*MonthIntervalType) String() string
*MonthIntervalType : DataType
*MonthIntervalType : FixedWidthDataType
*MonthIntervalType : github.com/polarsignals/frostdb/query/logicalplan.Named
*MonthIntervalType : expvar.Var
*MonthIntervalType : fmt.Stringer
Record as a term typically refers to a single row, but this type represents a batch of rows, known in Arrow parlance
as a RecordBatch. This alias is provided for backwards compatibility.
Deprecated: This is deprecated to avoid the confusion of the terminology where Record refers to a single row,
use [RecordBatch] instead.
StructType describes a nested type parameterized by an ordered sequence
of relative types, called its fields.(*StructType) Field(i int) Field FieldByName gets the field with the given name.
If there are multiple fields with the given name, FieldByName
returns the first such field. FieldIdx gets the index of the field with the given name.
If there are multiple fields with the given name, FieldIdx returns
the index of the first such field. FieldIndices returns indices of all fields with the given name, or nil. Fields method provides a copy of StructType fields
(so it can be safely mutated and will not result in updating the StructType). FieldsByName returns all fields with the given name.(*StructType) Fingerprint() string(*StructType) ID() Type(*StructType) Layout() DataTypeLayout(*StructType) Name() string(*StructType) NumFields() int(*StructType) String() string
*StructType : DataType
*StructType : NestedType
*StructType : github.com/polarsignals/frostdb/query/logicalplan.Named
*StructType : expvar.Var
*StructType : fmt.Stringer
func StructOf(fs ...Field) *StructType
func (*MapType).ValueType() *StructType
func github.com/apache/arrow-go/v18/arrow/array.NewStructBuilder(mem memory.Allocator, dtype *StructType) *array.StructBuilder
BitWidth returns the number of bits required to store a single element of this data type in memory. Bytes returns the number of bytes required to store a single element of this data type in memory.( TemporalWithUnit) Fingerprint() string( TemporalWithUnit) ID() Type( TemporalWithUnit) Layout() DataTypeLayout Name is name of the data type.( TemporalWithUnit) String() string( TemporalWithUnit) TimeUnit() TimeUnit
*DurationType
*Time32Type
*Time64Type
*TimestampType
TemporalWithUnit : DataType
TemporalWithUnit : FixedWidthDataType
TemporalWithUnit : github.com/polarsignals/frostdb/query/logicalplan.Named
TemporalWithUnit : expvar.Var
TemporalWithUnit : fmt.Stringer
TimestampType is encoded as a 64-bit signed integer since the UNIX epoch (2017-01-01T00:00:00Z).
The zero-value is a second and time zone neutral. In Arrow semantics, time zone neutral does not
represent a physical point in time, but rather a "wall clock" time that only has meaning within
the context that produced it. In Go, time.Time can only represent instants; there is no notion
of "wall clock" time. Therefore, time zone neutral timestamps are represented as UTC per Go
conventions even though the Arrow type itself has no time zone.TimeZonestringUnitTimeUnit BitWidth returns the number of bits required to store a single element of this data type in memory.(*TimestampType) Bytes() int ClearCachedLocation clears the cached time.Location object in the type.
This should be called if you change the value of the TimeZone after having
potentially called GetZone.(*TimestampType) Fingerprint() string GetToTimeFunc returns a function for converting an arrow.Timestamp value into a
time.Time object with proper TimeZone and precision. If the TimeZone is invalid
this will return an error. It calls GetZone to get the timezone for consistency. GetZone returns a *time.Location that represents the current TimeZone member
of the TimestampType. If it is "", "UTC", or "utc", you'll get time.UTC.
Otherwise it must either be a valid tzdata string such as "America/New_York"
or of the format +HH:MM or -HH:MM indicating an absolute offset.
The location object will be cached in the TimestampType for subsequent calls
so if you change the value of TimeZone after calling this, make sure to call
ClearCachedLocation.(*TimestampType) ID() Type(*TimestampType) Layout() DataTypeLayout(*TimestampType) Name() string(*TimestampType) String() string(*TimestampType) TimeUnit() TimeUnit
*TimestampType : DataType
*TimestampType : FixedWidthDataType
*TimestampType : TemporalWithUnit
*TimestampType : github.com/polarsignals/frostdb/query/logicalplan.Named
*TimestampType : expvar.Var
*TimestampType : fmt.Stringer
func github.com/apache/arrow-go/v18/arrow/array.NewTimestampBuilder(mem memory.Allocator, dtype *TimestampType) *array.TimestampBuilder
Type Parameters:
T: ValueType TypedArray is an interface representing an Array of a particular type
allowing for easy propagation of generics( TypedArray[T]) Data() ArrayData DataType returns the type metadata for this instance. Get single value to be marshalled with `json.Marshal` IsNull returns true if value at index is null.
NOTE: IsNull will panic if NullBitmapBytes is not empty and 0 > i ≥ Len. IsValid returns true if value at index is not null.
NOTE: IsValid will panic if NullBitmapBytes is not empty and 0 > i ≥ Len. Len returns the number of elements in the array.( TypedArray[T]) MarshalJSON() ([]byte, error) NullBitmapBytes returns a byte slice of the validity bitmap. NullN returns the number of null values in the array. Release decreases the reference count by 1.
Release may be called simultaneously from multiple goroutines.
When the reference count goes to zero, the memory is freed. Retain increases the reference count by 1.
Retain may be called simultaneously from multiple goroutines.( TypedArray[T]) String() string( TypedArray[T]) Value(int) T ValueStr returns the value at index as a string.
TypedArray : Array[T]
TypedArray : github.com/apache/arrow-go/v18/arrow/scalar.Releasable
TypedArray : github.com/goccy/go-json.Marshaler
TypedArray : encoding/json.Marshaler
TypedArray : expvar.Var
TypedArray : fmt.Stringer
func github.com/apache/arrow-go/v18/arrow/array.NewDictWrapper[T](dict *array.Dictionary) (TypedArray[T], error)
func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantArray).Metadata() TypedArray[[]byte]
func github.com/apache/arrow-go/v18/arrow/extensions.(*VariantArray).UntypedValues() TypedArray[[]byte]
TypeEqualOption is a functional option type used for configuring type
equality checks.
func CheckMetadata() TypeEqualOption
func TypeEqual(left, right DataType, opts ...TypeEqualOption) bool
UintType is a type constraint for raw values represented as unsigned
integer types by We aren't just using constraints.Unsigned
because we don't want to include the raw `uint` type here whose size
changes based on the architecture (uint32 on 32-bit architectures and
uint64 on 64-bit architectures). We also don't want to include uintptr
UnionType is an interface to encompass both Dense and Sparse Union types.
A UnionType is a nested type where each logical value is taken
from a single child. A buffer of 8-bit type ids (typed as UnionTypeCode)
indicates which child a given logical value is to be taken from. This is
represented as the "child id" or "child index", which is the index into the
list of child fields for a given child. ChildIDs returns a slice of ints to map UnionTypeCode values to
the index in the Fields that represents the given Type. It is
initialized with all values being InvalidUnionChildID (-1)
before being populated based on the TypeCodes and fields of the type.
The field for a given type can be retrieved by Fields()[ChildIDs()[typeCode]] Fields method provides a copy of NestedType fields
(so it can be safely mutated and will not result in updating the NestedType).( UnionType) Fingerprint() string( UnionType) ID() Type( UnionType) Layout() DataTypeLayout MaxTypeCode returns the value of the largest TypeCode in the list of typecodes
that are defined by this Union type Mode returns either SparseMode or DenseMode depending on the current
concrete data type. Name is name of the data type. NumFields provides the number of fields without allocating.( UnionType) String() string TypeCodes returns the list of available type codes for this union type
which will correspond to indexes into the ChildIDs slice to locate the
appropriate child. A union Array contains a buffer of these type codes
which indicate for a given index, which child has the value for that index.
*DenseUnionType
*SparseUnionType
UnionType : DataType
UnionType : NestedType
UnionType : github.com/polarsignals/frostdb/query/logicalplan.Named
UnionType : expvar.Var
UnionType : fmt.Stringer
func UnionOf(mode UnionMode, fields []Field, typeCodes []UnionTypeCode) UnionType
func github.com/apache/arrow-go/v18/arrow/array.Union.UnionType() UnionType
UnionTypeCode is an alias to int8 which is the type of the ids
used for union arrays.
ValueType is a generic constraint for valid Arrow primitive types
ViewHeader is a variable length string (utf8) or byte slice with
a 4 byte prefix and inline optimization for small values (12 bytes
or fewer). This is similar to Go's standard string but limited by
a length of Uint32Max and up to the first four bytes of the string
are copied into the struct. This prefix allows failing comparisons
early and can reduce CPU cache working set when dealing with short
strings.
There are two situations:
Entirely inlined string data
|----|------------|
^ ^
| |
size inline string data, zero padded
Reference into buffer
|----|----|----|----|
^ ^ ^ ^
| | | |
size prefix buffer index and offset to out-of-line portion
Adapted from TU Munich's UmbraDB [1], Velox, DuckDB.(*ViewHeader) BufferIndex() int32(*ViewHeader) BufferOffset() int32(*ViewHeader) Equals(buffers []*memory.Buffer, other *ViewHeader, otherBuffers []*memory.Buffer) bool(*ViewHeader) InlineBytes() (data []byte)(*ViewHeader) InlineString() (data string)(*ViewHeader) IsInline() bool(*ViewHeader) Len() int(*ViewHeader) Prefix() [4]byte(*ViewHeader) SetBytes(data []byte) int(*ViewHeader) SetIndexOffset(bufferIndex, offset int32)(*ViewHeader) SetString(data string) int
func github.com/apache/arrow-go/v18/arrow/array.(*BinaryView).ValueHeader(i int) *ViewHeader
func github.com/apache/arrow-go/v18/arrow/array.(*StringView).ValueHeader(i int) *ViewHeader
func github.com/apache/arrow-go/v18/arrow/array.ViewLike.ValueHeader(int) *ViewHeader
func (*ViewHeader).Equals(buffers []*memory.Buffer, other *ViewHeader, otherBuffers []*memory.Buffer) bool
Package-Level Functions (total 74)
CheckMetadata is an option for TypeEqual that allows checking for metadata
equality besides type equality. It only makes sense for types with metadata.
Date32FromTime returns a Date32 value from a time object
Date64FromTime returns a Date64 value from a time object
DenseUnionFromArrays enables creating a union type from a list of Arrays,
field names, and type codes. len(fields) should be either 0 or equal to len(children).
len(codes) should also be either 0, or equal to len(children).
If len(fields) == 0, then the fields will be named numerically as "0", "1", "2"...
and so on. If len(codes) == 0, then the type codes will be constructed as
[0, 1, 2, ..., n].
DenseUnionOf is equivalent to UnionOf(arrow.DenseMode, fields, typeCodes),
constructing a DenseUnionType from a list of fields and type codes.
If len(fields) != len(typeCodes) this will panic. They are allowed to be
of length 0.
FixedSizeListOf returns the list type with element type t.
For example, if t represents int32, FixedSizeListOf(10, t) represents [10]int32.
FixedSizeListOf panics if t is nil or invalid.
FixedSizeListOf panics if n is <= 0.
NullableElem defaults to true
FixedSizeListOfNonNullable is like FixedSizeListOf but NullableElem defaults to false
indicating that the child type should be marked as non-nullable.
Type Parameters:
T: FixedWidthType | ViewHeader GetBytes reinterprets a slice of T to a slice of bytes.
Type Parameters:
T: FixedWidthType | ViewHeader GetData reinterprets a slice of bytes to a slice of T.
NOTE: the buffer's length must be a multiple of Sizeof(T).
Type Parameters:
T: NumericType | bool | string | []byte | float16.Num GetDataType returns the appropriate DataType for the given type T
only for non-parametric types. This uses a map and reflection internally
so don't call this in a tight loop, instead call this once and then use
a closure with the result.
GetExtensionType retrieves and returns the extension type of the given name
from the global extension type registry. If the type isn't found it will return
nil. This function is safe to call from multiple goroutines concurrently.
Type Parameters:
T: int32 | int64 GetOffsets reinterprets the data.Buffers()[i] to a slice of T with len=data.Len()+1.
NOTE: the buffer's length must be a multiple of Sizeof(T).
Type Parameters:
T: NumericType | bool | string GetType returns the appropriate Type type T, only for non-parametric
types. This uses a map and reflection internally so don't call this in
a tight loop, instead call it once and then use a closure with the result.
Type Parameters:
T: FixedWidthType GetValues reinterprets the data.Buffers()[i] to a slice of T with len=data.Len().
If the buffer is nil, nil will be returned.
NOTE: the buffer's length must be a multiple of Sizeof(T).
LargeListOf returns the list type with element type t.
For example, if t represents int32, LargeListOf(t) represents []int32.
LargeListOf panics if t is nil or invalid. NullableElem defaults to true
LargeListOfNonNullable is like ListOf but NullableElem defaults to false, indicating
that the child type should be marked as non-nullable.
LargeListViewOf returns the list-view type with element type t.
For example, if t represents int32, LargeListViewOf(t) represents []int32.
LargeListViewOf panics if t is nil or invalid. NullableElem defaults to true
LargeListViewOfNonNullable is like LargeListViewOf but NullableElem defaults
to false, indicating that the child type should be marked as non-nullable.
ListOf returns the list type with element type t.
For example, if t represents int32, ListOf(t) represents []int32.
ListOf panics if t is nil or invalid. NullableElem defaults to true
ListOfNonNullable is like ListOf but NullableElem defaults to false, indicating
that the child type should be marked as non-nullable.
ListViewOf returns the list-view type with element type t.
For example, if t represents int32, ListViewOf(t) represents []int32.
ListViewOf panics if t is nil or invalid. NullableElem defaults to true
NarrowestDecimalType constructs the smallest decimal type that can represent
the requested precision. An error is returned if the requested precision
cannot be represented (prec <= 0 || prec > 76).
For reference:
prec in [ 1, 9] => Decimal32Type
prec in [10, 18] => Decimal64Type
prec in [19, 38] => Decimal128Type
prec in [39, 76] => Decimal256Type
NewChunked returns a new chunked array from the slice of arrays.
NewChunked panics if the chunks do not have the same data type.
NewColumn returns a column from a field and a chunked data array.
NewColumn panics if the field's data type is inconsistent with the data type
of the chunked data array.
NewColumnFromArr is a convenience function to create a column from
a field and a non-chunked array.
This provides a simple mechanism for bypassing the middle step of
constructing a Chunked array of one and then releasing it because
of the ref counting.
RegisterExtensionType registers the provided ExtensionType by calling ExtensionName
to use as a Key for registering the type. If a type with the same name is already
registered then this will return an error saying so, otherwise it will return nil
if successful registering the type.
This function is safe to call from multiple goroutines simultaneously.
SparseUnionFromArrays enables creating a union type from a list of Arrays,
field names, and type codes. len(fields) should be either 0 or equal to len(children).
len(codes) should also be either 0, or equal to len(children).
If len(fields) == 0, then the fields will be named numerically as "0", "1", "2"...
and so on. If len(codes) == 0, then the type codes will be constructed as
[0, 1, 2, ..., n].
SparseUnionOf is equivalent to UnionOf(arrow.SparseMode, fields, typeCodes),
constructing a SparseUnionType from a list of fields and type codes.
If len(fields) != len(typeCodes) this will panic. They are allowed to be
of length 0.
StructOf returns the struct type with fields fs.
StructOf panics if there is a field with an invalid DataType.
Time32FromString parses a string to return a Time32 value in the given unit,
unit needs to be only seconds or milliseconds and the string should be in the
form of HH:MM or HH:MM:SS[.zzz] where the fractions of a second are optional.
Time64FromString parses a string to return a Time64 value in the given unit,
unit needs to be only microseconds or nanoseconds and the string should be in the
form of HH:MM or HH:MM:SS[.zzzzzzzzz] where the fractions of a second are optional.
TimestampFromString parses a string and returns a timestamp for the given unit
level.
The timestamp should be in one of the following forms, [T] can be either T
or a space, and [.zzzzzzzzz] can be either left out or up to 9 digits of
fractions of a second.
YYYY-MM-DD
YYYY-MM-DD[T]HH
YYYY-MM-DD[T]HH:MM
YYYY-MM-DD[T]HH:MM:SS[.zzzzzzzz]
You can also optionally have an ending Z to indicate UTC or indicate a specific
timezone using ±HH, ±HHMM or ±HH:MM at the end of the string.
TimestampFromStringInLocation is like TimestampFromString, but treats the time instant
as if it were in the provided timezone before converting to UTC for internal representation.
TimestampFromTime allows converting time.Time to Timestamp
TypeEqual checks if two DataType are the same, optionally checking metadata
equality for STRUCT types.
TypesToString is a convenience function to create a list of types
which are comma delimited as a string
UnionOf returns an appropriate union type for the given Mode (Sparse or Dense),
child fields, and type codes. len(fields) == len(typeCodes) must be true, or else
this will panic. len(fields) can be 0.
UnregisterExtensionType removes the type with the given name from the registry
causing any messages with that type which come in to be expressed with their
metadata and underlying type instead of the extension type that isn't known.
This function is safe to call from multiple goroutines simultaneously.
The pages are generated with Goldsv0.8.2. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.