ArrayKernelExec is an alias definition for a kernel's execution function.
This is used for both stateless and stateful kernels. If a kernel
depends on some execution state, it can be accessed from the KernelCtx
object, which also contains the context.Context object which can be
used for shortcircuiting by checking context.Done / context.Err.
This allows kernels to control handling timeouts or cancellation of
computation.
ArraySpan is a light-weight, non-owning version of arrow.ArrayData
for more efficient handling with computation and engines. We use
explicit go Arrays to define the buffers and some scratch space
for easily populating and shifting around pointers to memory without
having to worry about and deal with retain/release during calculations.Buffers[3]BufferSpanChildren[]ArraySpanLenint64Nullsint64Offsetint64 Scratch is a holding spot for things such as
offsets or union type codes when converting from scalarsTypearrow.DataType Dictionary returns a pointer to the array span for the dictionary which
we will always place as the first (and only) child if it exists. FillFromScalar populates this ArraySpan as if it were a 1 length array
with the single value equal to the passed in Scalar. GetBuffer returns the buffer for the requested index. If this buffer
is owned by another array/arrayspan the Owning buffer is returned,
otherwise if this slice has no owning buffer, we call NewBufferBytes
to wrap it as a memory.Buffer. Can also return nil if there is no
buffer in this index. MakeArray is a convenience function for calling array.MakeFromData(a.MakeData()) MakeData generates an arrow.ArrayData object for this ArraySpan,
properly updating the buffer ref count if necessary.(*ArraySpan) MayHaveNulls() bool NumBuffers returns the number of expected buffers for this type if an error is encountered, call Release on a preallocated span
to ensure it releases any self-allocated buffers, it will
not call release on buffers it doesn't own (SelfAlloc != true)(*ArraySpan) SetDictionary(span *ArraySpan) SetMembers populates this ArraySpan from the given ArrayData object.
As this is a non-owning reference, the ArrayData object must not
be fully released while this ArraySpan is in use, otherwise any buffers
referenced will be released too SetSlice updates the offset and length of this ArraySpan to refer to
a specific slice of the underlying buffers. TakeOwnership is like SetMembers only this takes ownership of
the buffers by calling Retain on them so that the passed in
ArrayData can be released without negatively affecting this
ArraySpan UpdateNullCount will count the bits in the null bitmap and update the
number of nulls if the current null count is unknown, otherwise it just
returns the value of a.Nulls
func (*ArraySpan).Dictionary() *ArraySpan
func FillZeroLength(dt arrow.DataType, span *ArraySpan)
func GetSpanOffsets[T](span *ArraySpan, i int) []T
func GetSpanValues[T](span *ArraySpan, i int) []T
func NewBoolIter(arr *ArraySpan) ArrayIter[bool]
func NewFSBIter(arr *ArraySpan) ArrayIter[[]byte]
func NewPrimitiveIter[T](arr *ArraySpan) ArrayIter[T]
func NewVarBinaryIter[OffsetT](arr *ArraySpan) ArrayIter[[]byte]
func (*ArraySpan).SetDictionary(span *ArraySpan)
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.DispatchIsIn(state kernels.lookupState, in *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.GetTakeIndices(mem memory.Allocator, filter *ArraySpan, nullSelect kernels.NullSelectionBehavior) (arrow.ArrayData, error)
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.ShiftTime[InT, OutT](ctx *KernelCtx, op arrow.TimestampConvertOp, factor int64, input, output *ArraySpan) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.AndNotOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.AndNotOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.AndNotOpKernel.CallScalarRight(ctx *KernelCtx, left *ArraySpan, right scalar.Scalar, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.AndOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.AndOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.HashState.Append(*KernelCtx, *ArraySpan) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneAndNotOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneAndNotOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneAndNotOpKernel.CallScalarRight(ctx *KernelCtx, left *ArraySpan, right scalar.Scalar, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneAndOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneAndOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneOrOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.KleeneOrOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.OrOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.OrOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.(*SetLookupState)[T].AddArrayValueSet(data *ArraySpan, startIdx int64) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.SimpleBinaryKernel.Call(*KernelCtx, *ArraySpan, *ArraySpan, *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.SimpleBinaryKernel.Call(*KernelCtx, *ArraySpan, *ArraySpan, *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.SimpleBinaryKernel.CallScalarLeft(*KernelCtx, scalar.Scalar, *ArraySpan, *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.SimpleBinaryKernel.CallScalarRight(*KernelCtx, *ArraySpan, scalar.Scalar, *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.XorOpKernel.Call(ctx *KernelCtx, left, right *ArraySpan, out *ExecResult) error
func github.com/apache/arrow-go/v18/arrow/compute/internal/kernels.XorOpKernel.CallScalarLeft(ctx *KernelCtx, left scalar.Scalar, right *ArraySpan, out *ExecResult) error
BufferSpan is a lightweight Buffer holder for ArraySpans that does not
take ownership of the underlying memory.Buffer at all or could be
used to reference raw byte slices instead. Buf should be the byte slice representing this buffer, if this is
nil then this bufferspan should be considered empty. Owner should point to an underlying parent memory.Buffer if this
memory is owned by a different, existing, buffer. Retain is not
called on this buffer, so it must not be released as long as
this BufferSpan refers to it. SelfAlloc tracks whether or not this bufferspan is the only owner
of the Owning memory.Buffer. This happens when preallocating
memory or if a kernel allocates it's own buffer for a result.
In these cases, we have to know so we can properly maintain the
refcount if this is later turned into an ArrayData object. SetBuffer sets the given buffer into this BufferSpan and marks
SelfAlloc as false. This should be called when setting a buffer
that is externally owned/created. WrapBuffer wraps this bufferspan around a buffer and marks
SelfAlloc as true. This should be called when setting a buffer
that was allocated as part of an execution rather than just
re-using an existing buffer from an input array.
ChunkedExec is the signature for executing a stateful vector kernel
against a ChunkedArray input. It is optional
ExecResult is the result of a kernel execution and should be populated
by the execution functions and/or a kernel. For now we're just going to
alias an ArraySpan.
InputKind is an enum representing the type of Input matching
that will be done. Either accepting any type, an exact specific type
or using a TypeMatcher.
const InputAny
const InputExact
const InputUseMatcher
ResolveKind defines the way that a particular OutputType resolves
its type. Either it has a fixed type to resolve to or it contains
a Resolver which will compute the resolved type based on
the input types.
const ResolveComputed
const ResolveFixed
BinaryLike returns a TypeMatcher that will match Binary or String
DurationTypeUnit returns a TypeMatcher that will match only
a Duration datatype with the specified TimeUnit.
FillZeroLength fills an ArraySpan with the appropriate information for
a Zero Length Array of the provided type.
FixedSizeBinaryLike returns a TypeMatcher that will match FixedSizeBinary
or Decimal128/256
GetAllocator retrieves the allocator from the context, or returns
memory.DefaultAllocator if there was no allocator in the provided
context.
Type Parameters:
T: int32 | int64 GetSpanOffsets is like GetSpanValues, except it is only for int32
or int64 and adds the additional 1 expected value for an offset
buffer (ie. len(output) == span.Len+1)
Type Parameters:
T: arrow.FixedWidthType GetSpanValues returns a properly typed slice by reinterpreting
the buffer at index i using unsafe.Slice. This will take into account
the offset of the given ArraySpan.
NewScalarKernel constructs a new kernel for scalar execution, constructing
a KernelSignature with the provided input types and output type, and using
the passed in execution implementation and initialization function.
NewScalarKernelWithSig is a convenience when you already have a signature
to use for constructing a kernel. It's equivalent to passing the components
of the signature (input and output types) to NewScalarKernel.
NewVectorKernel constructs a new kernel for execution of vector functions,
which take into account more than just the individual scalar values
of its input. Output of a vector kernel may be a different length
than its inputs.
NewVectorKernelWithSig is a convenience function for creating a kernel
when you already have a signature constructed.
Type Parameters:
T: any OptionsInit should be used in the case where a KernelState is simply
represented with a specific type by value (instead of pointer).
This will initialize the KernelState as a value-copied instance of
the passed in function options argument to ensure separation
and allow the kernel to manipulate the options if necessary without
any negative consequences since it will have its own copy of the options.
Primitive returns a TypeMatcher that will match any type that arrow.IsPrimitive
returns true for.
PromoteExecSpanScalars promotes the values of the passed in ExecSpan
from scalars to Arrays of length 1 for each value.
The kernel is responsible for allocating its own data buffer
for fixed-width output types.
For data types that support pre-allocation (fixed-width), the
kernel expects to be provided a pre-allocated buffer to write into.
Non-fixed-width types must always allocate their own buffers.
The allocation is made for the same length as the execution batch,
so vector kernels yielding differently sized outputs should not
use this.
It is valid for the data to not be preallocated but the validity
bitmap is (or is computed using intersection).
For variable-size output types like Binary or String, or for nested
types, this option has no effect.
Kernel will allocate and set the validity bitmap of the output
Kernel expects a pre-allocated buffer to write the result bitmap
into.
Compute the output validity bitmap by intersection the validity
bitmaps of the arguments using bitwise-and operations. This means
that values in the output are valid/non-null only if the corresponding
values in all input arguments were valid/non-null. Kernels generally
do not have to touch the bitmap afterwards, but a kernel's exec function
is permitted to alter the bitmap after the null intersection is computed
if necessary.
kernel output is never null and a validity bitmap doesn't need to
be allocated
The pages are generated with Goldsv0.8.2. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.
