package rcpg
Import Path
github.com/freeeve/gochickpeas/rcpg (on go.dev)
Dependency Relation
imports 9 packages, and imported by one package
Involved Source Files
Block-lazy atom resolution over a SectionFetch: the first consumer-shaped
slice of the working-set machinery lazy.go defers. An AtomReader keeps a
small resident router (one byte offset per block of atoms) and range-fetches
one block per lookup, so a skeleton parse (SkeletonParseOptions) plus an
AtomReader serves traversal with a hot atom set without ever holding the
atom table resident. RCPG v1 stores atoms as contiguous length-prefixed
strings with no in-file offset index, so the router is discovered by
scanning length prefixes forward, incrementally, at most once per file.
column.go
cursor.go
errors.go
lazy.go
Package rcpg reads and writes RCPG, the RustyChickpeas graph file, and its
companion RRSR record store (subpackage rrsr). The byte layout is frozen by
FORMAT.md in the rustychickpeas repository; this implementation is
conformance-tested byte-for-byte against the Rust codec.
Quick sketch (all integers little-endian):
header (16 B): magic "RCPG" | version u16 | flags u16 | section_count u32 | reserved u32
directory: section_count x { id u32, reserved u32, offset u64, length u64 }
sections: at the directory offsets (relative to file start)
Section IDs: 1 atoms, 2 meta, 3 nodes, 4 relationships, 5 node columns,
6 relationship columns, 7 optional atom block index, 8 optional node
existence bitmap (sparse id spaces only -- absent means every
in-id-space id is a node, which is exactly true for dense writers).
Unknown section IDs are ignored on read so the format can grow; files
with a version above 1 are rejected.
section.go
Package-Level Type Names (total 25)
AtomBlockIndex is a parsed section-7 body: Offsets[k] is the byte offset,
relative to the atoms section body start, of atom k*BlockLen's length
prefix (entry 0 is always 4, past the section's count word).
BlockLen uint32
Offsets []uint64
AtomReader resolves atom ids on demand through a SectionFetch, without
materializing the atom table. Lookups route through a resident block
index (built incrementally by scanning the section's length prefixes up
to the highest block touched so far) and fetch + decode one block of
atoms, kept in a small FIFO cache. Structural bounds are validated during
the scan; UTF-8 validity is checked per decoded block, like the eager
decoder but only for atoms actually read.
An AtomReader is safe for concurrent use; lookups that miss the cache
serialize on the underlying fetch.
Atom resolves one atom id, fetching and decoding its block on a cache
miss. Ids at or past Count error (a conforming file never references
them), wrapping ErrCorrupt.
Count is the number of atoms in the table.
func NewAtomReader(f SectionFetch, dir *Directory) (*AtomReader, error)
BitmapEntry is one entry of a bitmap index: an atom mapped to a roaring
bitmap (label atom -> node IDs, or type atom -> outgoing-CSR positions).
Atom uint32
Bitmap *roaring.Bitmap
ByteRange is a half-open [Start, End) byte range within a file.
End uint64
Start uint64
func (*Directory).PlanSections(opts ParseOptions) []ByteRange
Column is one property column: a key atom and its storage.
Data ColumnData
Key uint32
ColumnData is one property column's storage. Dense columns are indexed by
ID (node ID, or outgoing-CSR position for rel columns); sparse columns are
(id, value) pairs sorted by ID. String values are atom IDs into
GraphSection.Atoms; atom 0 ("") in a dense str column means missing.
DenseBool
DenseF64
DenseI64
DenseStr
SparseBool
SparseF64
SparseI64
SparseStr
DenseBool is a dense bool column as stored: Len bits packed LSB-first
within each byte, ceil(Len/8) bytes.
Bits []byte
Len uint32
Get reports bit i, false when out of range.
Set sets bit i to v; i must be < Len.
DenseBool : ColumnData
func NewDenseBool(n uint32) DenseBool
DenseF64 is a dense float64 column; values are written as raw IEEE-754
bits, so NaN payloads and -0.0 survive round trips exactly.
DenseF64 : ColumnData
DenseI64 is a dense int64 column.
DenseI64 : ColumnData
DenseStr is a dense string column of value atoms.
DenseStr : ColumnData
Directory is the RCPG header + section directory — the small, resident
front matter of a file. Parse it from the leading bytes (16-byte header +
section_count x 24-byte entries), then PlanSections to get the byte ranges
a given ParseOptions needs, so the bodies can be range-fetched without
transferring the whole file.
Entries returns the directory entries in file order.
LoadPrefixLen is the smallest prefix length [0, len) that contains every
section needed under opts — fetch this many bytes and ParseLazy over a
MemoryFetch of them to load without the trailing (unwanted) sections.
Because the topology sections precede the property columns in the file, a
topology-only prefix omits the columns entirely; a full load returns the
file length.
PlanSections returns the byte ranges to fetch for opts: the topology
sections always, the property columns only when their option is set.
Unknown sections are skipped. Order follows the directory, so the ranges
are ascending and can be coalesced for batched fetches.
func ParseDirectory(front []byte) (*Directory, error)
func NewAtomReader(f SectionFetch, dir *Directory) (*AtomReader, error)
DirEntry is one section directory entry: (id, offset, length) in file
order.
ID uint32
Length uint64
Offset uint64
func (*Directory).Entries() []DirEntry
GraphSection is the complete decoded content of an RCPG file.
All string-ish values (labels, relationship types, property keys, string
property values) are atom IDs into Atoms; atom 0 is always the empty
string. A nil Version means the meta section carried no version string
(present-but-empty is distinct on disk).
AtomIndex is the optional section-7 block index over the atoms
section, present only when the file carried one: lazy readers route
an atom id straight to its block instead of scanning the section's
length prefixes. Nil on a hand-built graph; the writer derives it
from the atoms at encode time, so it carries no independent
information.
Atoms is the string table; index 0 is always "".
Existence is the optional section-8 node existence bitmap: the set
of ids that are actual nodes, for id spaces with gaps (builder-
chosen sparse ids). Nil means absent, in which case every
in-id-space id is a node -- exactly true for dense writers, whose
files stay byte-identical by never emitting the section.
InNbrs holds source node IDs, len = NRels.
InOffsets is sized like OutOffsets.
InTypes holds relationship type atoms, parallel to InNbrs.
LabelIndex maps label atom -> node IDs, sorted by label atom.
NNodes is the actual number of distinct nodes, NOT the CSR ID-space
size; see OutOffsets.
NRels is the number of relationships.
NodeColumns are node property columns, sorted by key atom.
OutNbrs holds destination node IDs, len = NRels.
OutOffsets has csr_id_space + 1 entries. IDs never added have empty
ranges, so the ID space can exceed NNodes under sparse IDs.
OutTypes holds relationship type atoms, parallel to OutNbrs.
RelColumns are relationship property columns, indexed by the
relationship's outgoing-CSR position.
TypeIndex maps type atom -> outgoing-CSR relationship positions,
sorted by type atom.
Version is the optional snapshot version string.
CSRIDSpace is the size of the CSR ID space (max node ID + 1 for non-empty
graphs).
InNeighbors returns the incoming neighbors (source node IDs) of node,
empty for IDs outside the ID space.
OutNeighbors returns the outgoing neighbors of node, empty for IDs outside
the ID space.
func Parse(b []byte) (*GraphSection, error)
func ParseLazy(f SectionFetch, opts ParseOptions) (*GraphSection, error)
func ParseWith(b []byte, opts ParseOptions) (*GraphSection, error)
func github.com/freeeve/gochickpeas.(*Snapshot).ToGraphSection() *GraphSection
func Write(g *GraphSection, w io.Writer) error
func WriteWith(g *GraphSection, w io.Writer, opts WriteOptions) error
func github.com/freeeve/gochickpeas.FromGraphSection(section *GraphSection) *chickpeas.Snapshot
MemoryFetch is a SectionFetch over an in-memory RCPG slice, returning
sub-slices with no copy — the lazy-load equivalent of eager Parse, useful
for embedding and for parity tests of the lazy path.
Fetch implements SectionFetch.
MemoryFetch : SectionFetch
ParseOptions controls which optional sections a reader materializes.
Skipped sections cost nothing to parse and nothing in resident memory,
regardless of whether the file contains them.
NodeColumns bool
RelColumns bool
SkipAtoms leaves the atom table out of the parse: GraphSection.Atoms
stays nil and ParseLazy never fetches the atoms section. Labels, rel
types, and property keys remain raw atom ids; resolve them on demand
through an AtomReader over the same SectionFetch.
func DefaultParseOptions() ParseOptions
func SkeletonParseOptions() ParseOptions
func TopologyOnlyParseOptions() ParseOptions
func ParseLazy(f SectionFetch, opts ParseOptions) (*GraphSection, error)
func ParseWith(b []byte, opts ParseOptions) (*GraphSection, error)
func (*Directory).LoadPrefixLen(opts ParseOptions) uint64
func (*Directory).PlanSections(opts ParseOptions) []ByteRange
SectionFetch is a byte-range provider for lazy RCPG loading: it returns
[offset, offset+length) of an RCPG file. Implementations must return
exactly length bytes or an error. Callers plug in files (io.ReaderAt),
HTTP Range requests, or object-store reads.
( SectionFetch) Fetch(offset, length uint64) ([]byte, error)
MemoryFetch
func NewAtomReader(f SectionFetch, dir *Directory) (*AtomReader, error)
func ParseLazy(f SectionFetch, opts ParseOptions) (*GraphSection, error)
SparseBool is a sparse bool column, sorted by ID.
SparseBool : ColumnData
SparseF64 is a sparse float64 column, sorted by ID.
SparseF64 : ColumnData
SparseI64 is a sparse int64 column, sorted by ID.
SparseI64 : ColumnData
SparseStr is a sparse string column, sorted by ID.
SparseStr : ColumnData
WriteOptions controls which optional sections a writer emits. Topology
(atoms, meta, nodes, relationships) is always written; property columns
are optional so large graphs can ship a lean traversal-only file and leave
per-node data to a range-fetched record store.
AtomIndex emits the optional section-7 atom block index. Default
off, so existing byte-identity guarantees (and the vendored
conformance corpus) are unchanged unless a writer opts in.
Existence emits the optional section-8 node existence bitmap when
the graph carries one (sparse id spaces). Default off for the same
byte-identity reason as AtomIndex: the conformance corpus is the
cross-engine contract, and the section joins it in a coordinated
corpus bump once the Rust writer emits it too. Until a sparse
graph is written WITH this, its round trip keeps the legacy
every-in-space-id-exists presumption.
NodeColumns bool
RelColumns bool
func DefaultWriteOptions() WriteOptions
func TopologyOnlyWriteOptions() WriteOptions
func WriteWith(g *GraphSection, w io.Writer, opts WriteOptions) error
func github.com/freeeve/gochickpeas.(*Snapshot).WriteRCPGWith(w io.Writer, opts WriteOptions) error
Package-Level Functions (total 15)
DefaultParseOptions materializes every section.
DefaultWriteOptions emits every section.
FrontMatterLen is the number of front-matter bytes (header + directory)
for a file with sectionCount sections — what to fetch before the directory
can be parsed.
NewAtomReader opens block-lazy atom resolution over f, locating the atoms
section through dir and fetching only its 4-byte count header. Errors wrap
ErrCorrupt when the directory has no atoms section or the count cannot fit
the section.
NewDenseBool returns an all-false column of n bits.
Parse decodes RCPG bytes into a GraphSection, materializing all sections.
It never panics on malformed input; all structural problems surface as
errors wrapping ErrCorrupt (or ErrUnsupportedVersion for a future format
version).
ParseDirectory parses the header + full directory from front (which must
hold at least FrontMatterLen bytes). Body offsets are recorded but not
validated against any file length — the bodies live elsewhere and are
checked when fetched.
ParseLazy parses an RCPG file through a SectionFetch, transferring only
the sections opts needs. It reads the header, then the directory, then
fetches and decodes each wanted section body — so a topology-only load
over a range-capable transport never transfers the property columns.
Produces a GraphSection identical to ParseWith for the same opts.
ParseWith decodes RCPG bytes, materializing optional sections per opts.
Use TopologyOnlyParseOptions to keep property columns out of memory on
large graphs.
SectionCount reads the section count from a header prefix (at least 16
bytes), validating the magic and version. Use with FrontMatterLen to size
the directory fetch.
SkeletonParseOptions materializes the CSR skeleton only: no property
columns and no atom table. The lightest resident load for traversal-first
consumers that resolve atom ids lazily via AtomReader.
TopologyOnlyParseOptions ignores property column sections even when
present.
TopologyOnlyWriteOptions emits no property column sections.
Write serializes a graph to RCPG bytes, including all sections.
WriteWith serializes a graph to RCPG bytes, emitting optional sections per
opts. The output is deterministic: section order, and the order of bitmap
index entries and columns within their sections, follow the GraphSection
slices exactly (which the spec requires sorted by atom).
Package-Level Variables (total 3)
ErrCorrupt reports input bytes that are not a valid file of the expected
format. Every structural problem in a parse surfaces as an error wrapping
this sentinel; parsers never panic on malformed input.
ErrUnsupportedVersion reports a file that is a recognized format but a
newer, unsupported version.
Magic is the RCPG file magic.
Package-Level Constants (only one)
Version is the RCPG format version this package reads and writes.
 |
The pages are generated with Golds v0.8.7. (GOOS=linux GOARCH=amd64) Golds is a Go 101 project developed by Tapir Liu. PR and bug reports are welcome and can be submitted to the issue list. Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds. |