Source File
reader_properties.go
Belonging Package
github.com/apache/arrow-go/v18/parquet
// Licensed to the Apache Software Foundation (ASF) under one// or more contributor license agreements. See the NOTICE file// distributed with this work for additional information// regarding copyright ownership. The ASF licenses this file// to you under the Apache License, Version 2.0 (the// "License"); you may not use this file except in compliance// with the License. You may obtain a copy of the License at//// http://www.apache.org/licenses/LICENSE-2.0//// Unless required by applicable law or agreed to in writing, software// distributed under the License is distributed on an "AS IS" BASIS,// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.// See the License for the specific language governing permissions and// limitations under the License.package parquetimport ()// ReaderProperties are used to define how the file reader will handle buffering and allocating bufferstype ReaderProperties struct {alloc memory.Allocator// Default buffer size to utilize when reading chunks, when reading page// headers or other metadata, this buffer may be increased if necessary// to read in the necessary metadata. The value here is simply the default// initial BufferSize when reading a new chunk.BufferSize int64// create with NewFileDecryptionProperties if dealing with an encrypted fileFileDecryptProps *FileDecryptionProperties// If this is set to true, then the reader will use SectionReader to// just use the read stream when reading data. Otherwise we will buffer// the data we're going to read into memory first and then read that buffer.//// When accessing data from IO sources with higher latency, like S3, setting this// to false may improve performance by reading the entire row group at once rather// than sending multiple smaller IO requests. For IO streams with low latency, setting// this to true can optimize memory usage for the reader. Additionally, this can decrease// the amount of data retrieved when only needs to access small portions of the parquet file.BufferedStreamEnabled bool}type BufferedReader interface {Peek(int) ([]byte, error)Discard(int) (int, error)Outer() utils.ReaderBufferSize() intReset(utils.Reader)io.Reader}// NewReaderProperties returns the default Reader Properties using the provided allocator.//// If nil is passed for the allocator, then memory.DefaultAllocator will be used.func ( memory.Allocator) *ReaderProperties {if == nil {= memory.DefaultAllocator}return &ReaderProperties{, DefaultBufSize, nil, false}}// Allocator returns the allocator that the properties were initialized withfunc ( *ReaderProperties) () memory.Allocator { return .alloc }// GetStream returns a section of the underlying reader based on whether or not BufferedStream is enabled.//// If BufferedStreamEnabled is true, it creates an io.SectionReader, otherwise it will read the entire section// into a buffer in memory and return a bytes.NewReader for that buffer.func ( *ReaderProperties) ( io.ReaderAt, , int64) (BufferedReader, error) {if .BufferedStreamEnabled {return utils.NewBufferedReader(io.NewSectionReader(, , ), int(.BufferSize)), nil}:= make([]byte, ), := .ReadAt(, )if != nil {return nil, fmt.Errorf("parquet: tried reading from file, but got error: %w", )}if != int() {return nil, fmt.Errorf("parquet: tried reading %d bytes starting at position %d from file but only got %d", , , )}return utils.NewByteReader(), nil}
![]() |
The pages are generated with Golds v0.8.2. (GOOS=linux GOARCH=amd64) Golds is a Go 101 project developed by Tapir Liu. PR and bug reports are welcome and can be submitted to the issue list. Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds. |