Involved Source Filesarena.gocanon.godecoder.gographview.gojsonld.gontriples.goquad.go Package rdf is a small, fast, dependency-free RDF toolkit. It provides the RDF
triple model (Term, Triple, Graph), parsers and serializers for the four common
serializations — RDF/XML, JSON-LD, Turtle and N-Triples — and a streaming
decoder for the line-based formats.
Two reading modes:
- Whole document: ParseRDFXML, ParseJSONLD, ParseTurtle and ParseNTriples take
a []byte and return a *Graph. Fast and convenient for inputs that fit in
memory. The line-based parsers have zero-copy variants (ParseNTriplesShared,
ParseNQuadsShared) that back terms with the caller's buffer instead of a
private copy — one input-sized allocation less, for callers that keep the
buffer immutable.
- Streaming: NewDecoder reads N-Triples, N-Quads, RDF/XML or Turtle from an
io.Reader one triple at a time in constant memory, for inputs too large to
materialize (e.g. the multi-gigabyte Library of Congress authority dumps).
JSON-LD is whole-document only.
A parsed N-Quads document is a Dataset. Reading one of its named graphs has two
modes: Dataset.Graph materializes the graph's triples as a Graph, while
Dataset.GraphView answers the same queries (the GraphQuery surface) over
positions into the dataset's own quads, copying no triple. Prefer the view for
read-only access at corpus scale, where the copy dominates the allocation
profile.
The parsers target the constructs real-world RDF uses rather than the whole of
each specification; see each parser's documentation for what is and isn't
handled (notably, relative-IRI resolution against a document base is not
performed). There are no third-party dependencies.rdfxml.goturtle.goturtle_lex.goturtle_write.go
Package-Level Type Names (total 12)
/* sort by: | */
Dataset holds quads — RDF statements each tagged with the graph they belong
to — in document order. It is the quad-level analogue of Graph and what
ParseNQuads returns; it is how provenance (which source a statement came
from) is carried.
Like [Graph.Triples], Quads is a list and may repeat a statement the document
stated twice; see the Graph doc for why parsing preserves that. Quads are in document order and may repeat; see the type doc. Add appends a quad. Canonical returns the dataset serialized as canonical N-Quads per the W3C RDF
Dataset Canonicalization algorithm RDFC-1.0 (URDNA2015): blank nodes are
relabeled to isomorphism-invariant _:c14nN identifiers and the quads are sorted,
so two datasets that differ only in statement order or blank-node labels produce
byte-identical output. It returns ErrCanonComplexity for a graph whose symmetry
exceeds the work budget. Graph returns the triples belonging to the given graph term as a Graph; pass a
zero-value term for the default graph. It copies one Triple per matching quad,
so a caller that only reads the result should use GraphView instead, which
answers the same queries over the dataset's quads without copying. GraphLen returns the number of statements in the given graph term; pass a
zero-value term for the default graph. It reads a cached per-graph count, so
after the first call on a dataset it costs no scan — asking about several
graphs, or about a graph that turns out to be empty, is one pass in total
rather than one pass per graph. GraphView returns a read-only, zero-copy view of the given graph term; pass a
zero-value term for the default graph. It is the allocation-free counterpart
to Graph, which materializes the same statements as a []Triple copy. Graphs returns the distinct graph terms present in the dataset, in first-seen
order — the set of provenance sources. It reads the same cached counts GraphLen
does, which record each graph on first sight, so repeated calls cost no scan. HasGraph reports whether the dataset carries any statement in the given graph
term. Like GraphLen it answers from the cached counts rather than scanning, so
skipping an absent graph is free. NQuads serializes the dataset as an N-Quads document.
func Canonicalize(d *Dataset) (*Dataset, error)
func ParseNQuads(data []byte) (*Dataset, error)
func ParseNQuadsShared(data []byte) (*Dataset, error)
func Canonicalize(d *Dataset) (*Dataset, error)
Decoder streams RDF statements from an io.Reader one triple at a time, in
constant memory — for inputs far too large to materialize into a Graph
(multi-gigabyte dumps such as the LC authority files). Each returned triple owns
its strings, so it is safe to retain after the next call.
JSON-LD is not streamable here (its document must be materialized); use
ParseJSONLD for that. All returns an iterator over the remaining triples, ending at the first error
(io.EOF, the normal end, is not surfaced). Breaking out of the loop stops the
producer:
for tr := range dec.All() { ... }
Use Decode directly when a non-EOF read error must be observed. AllQuads returns an iterator over the remaining statements as quads,
preserving N-Quads graph terms. Like All, it ends at the first error and
breaking out of the loop stops the producer:
for q := range dec.AllQuads() { ... } Close stops a streaming decoder early, releasing its producer goroutine. It is a
no-op for the line-based formats and after the stream is exhausted. Decode returns the next triple, or io.EOF when the input is exhausted. Any
fourth (graph) term on an N-Quads line is dropped; use DecodeQuad to keep it.
Blank and comment lines are skipped; a malformed one is a *SyntaxError naming
the line, unless [Decoder.SkipMalformed] was set. io.EOF therefore means the
input ended, not that it ended and some of it was unreadable. DecodeQuad returns the next statement as a quad, preserving the graph term of
an N-Quads line; the triple-only formats (N-Triples, RDF/XML, Turtle) yield a
zero-value graph term (the default graph). It returns io.EOF at end of input. SkipMalformed makes the decoder skip lines it cannot parse instead of failing
on them, and returns the decoder so it can be chained onto [NewDecoder]. It
affects only the line-based formats (N-Triples, N-Quads); RDF/XML and Turtle
have always reported a syntax error.
The default is to fail. A parser that skips what it cannot read turns a
truncated dump into a smaller, well-formed graph, and the caller has no way to
tell. Opt in only where the input is known to carry noise that is safe to drop
-- and where a short read is not a lie you would ship.
*Decoder : github.com/prometheus/common/expfmt.Closer
*Decoder : io.Closer
func NewDecoder(r io.Reader, format Format) *Decoder
func (*Decoder).SkipMalformed(skip bool) *Decoder
Encoder serializes a sequence of graphs into one document — N-Triples,
N-Quads or Turtle — keeping blank-node labels unique across the graphs it
writes. Blank-node labels are scoped to the whole document, so a single
Encoder must be reused for every graph written to one stream; a fresh encoder
per graph would let graphs that each number their blanks from scratch (such as
one BIBFRAME graph per record) collide and merge. It is the serialization
counterpart to Decoder; the whole-graph Graph.NTriples, Graph.NQuads and
Graph.Turtle methods each use a fresh Encoder internally. AppendNQuads appends every triple of g to b as N-Quads tagged with the graph
term — g's named graph / provenance. A zero-value graph term writes the
default graph (plain N-Triples). Each call is a fresh blank-node scope, so
graphs that label their blanks from scratch never merge, while their output
labels stay unique across the whole document. AppendNTriples appends g's triples to b as N-Triples in a fresh blank-node
scope — N-Triples being N-Quads restricted to the default graph. AppendQuad appends one quad to b as an N-Quads line. A default-graph quad
(zero-value G) is written as a three-term N-Triples line. AppendTurtle appends g's triples to b as a Turtle body (no @prefix header),
grouped by subject, in a fresh blank-node scope. It groups without a
per-subject map: subjects are ranked by first appearance, then a stable sort
of triple indices by that rank makes each subject's triples contiguous in
document order — turning thousands of small allocations into a handful.
Graph holds the triples of one RDF graph in document order, with simple
lookup helpers built on first use. Term is comparable, so it indexes by
subject term directly — no key strings.
Triples is a list, not a set. RDF 1.1 defines a graph as "a set of RDF
triples", but a serialization is free to state the same triple more than
once, and real ones do: LC's own marc2bibframe2 output repeats a shared
agency or vocabulary node under every property that references it, so one of
their N-Triples files carries 449 lines for 389 distinct triples. Parsing
preserves what the document said, which keeps the parse allocation-light and
keeps redundant statements visible to a caller who cares. It also means
len(g.Triples) can exceed the triple count a set-backed parser such as rdflib
reports for the same document.
The query helpers below hide that: [Graph.Objects] and [Graph.SubjectsOfType]
return each distinct answer once. Callers who want set semantics over the
triples themselves can call [Graph.Dedupe], and [Graph.Canonical] already
collapses duplicates on its way to canonical form. Triples are in document order and may repeat; see the type doc. Add appends a triple. Canonical returns the graph as canonical N-Quads (its triples in the default
graph), per RDFC-1.0. See Dataset.Canonical. Dedupe removes duplicate triples in place, keeping the first occurrence of
each and preserving document order among the survivors, and reports how many
it removed. It gives the graph the set semantics RDF 1.1 ascribes to one, at
the cost of a hash of every triple — which is why parsing does not do it. HasType reports whether the subject has rdf:type typeIRI. It scans the index
without allocating. Literal returns the value of the subject's first literal object for the
predicate, or false. It scans the index without allocating an intermediate
slice, serving the frequent single-value field reads. NQuads serializes the graph's triples as N-Quads, each tagged with the given
graph term (its named graph / provenance). A zero-value graph term produces
plain N-Triples. NTriples serializes the graph as N-Triples. Object returns the first object for (subject, predicate), or false. It scans the
index without allocating an intermediate slice. Objects returns the distinct objects of every triple with the given subject
and predicate IRI, in document order. A document that states the same triple
twice yields that object once: the answer is a property's value set, not a
count of how often the document repeated itself. ObjectsWithRepeats returns the objects of every triple with the given subject
and predicate IRI, statement for statement, in document order -- including a
term the document stated more than once.
This is the raw list view, and it exists because [Graph.Triples] is a list.
Prefer [Graph.Objects]: a repeated statement carries no information in RDF, so
a caller that treats its multiplicity as meaningful is reading something the
abstract syntax does not say, and will disagree with any set-backed store that
touches the same document. The one honest use is inspecting a serialization as
written -- counting redundant statements, or round-tripping a positional
encoding that a conformant parser would already have flattened. SubjectsOfType returns every subject with rdf:type typeIRI, in document order
(deduplicated). Turtle serializes the graph as Turtle, declaring the given namespace prefixes
(prefix label -> namespace IRI) and compacting IRIs against them. Triples are
grouped by subject in first-seen order, using `a` for rdf:type. TurtleBody serializes the triples grouped by subject, without a prefix header.
*Graph : GraphQuery
func ParseJSONLD(data []byte) (*Graph, error)
func ParseNTriples(data []byte) (*Graph, error)
func ParseNTriplesShared(data []byte) (*Graph, error)
func ParseRDFXML(data []byte) (*Graph, error)
func ParseTurtle(data []byte) (*Graph, error)
func (*Dataset).Graph(graph Term) *Graph
func (*Encoder).AppendNQuads(b []byte, g *Graph, graph Term) []byte
func (*Encoder).AppendNTriples(b []byte, g *Graph) []byte
func (*Encoder).AppendTurtle(b []byte, g *Graph, prefixes map[string]string) []byte
GraphView is a read-only view of the statements in one named graph of a
Dataset. It answers the Graph query surface without copying any triple: its
lazy index holds int32 positions into the dataset's Quads, the same shape as
Graph's subject index. Pass a zero-value graph term to view the default graph.
A view borrows the dataset, so the dataset must outlive it. Appending to the
dataset invalidates the view's index; the next query rebuilds it, so a view
stays correct across Add at the cost of one reindex. Views are not safe for
concurrent use with a writer, and the first query on a view builds the index,
so concurrent readers must either share a view whose index is already built
or hold one view each. Empty reports whether the view's graph holds no statements. Like Len it answers
from the dataset's cached counts, so a consumer that reads a graph only when it
is populated — an editorial overlay, say — pays nothing to skip an absent one. GraphTerm returns the graph term this view is scoped to. HasType reports whether the subject has rdf:type typeIRI in this graph. It
scans the index without allocating. Len returns the number of statements in the view's graph. It reads the
dataset's cached per-graph counts, so it costs no scan after the first call on
that dataset, however many graphs are asked about. Literal returns the value of the subject's first literal object for the
predicate in this graph, or false. It scans the index without allocating an
intermediate slice, serving the frequent single-value field reads. Object returns the first object for (subject, predicate) in this graph, or
false. It scans the index without allocating an intermediate slice. Objects returns the distinct objects of every statement in this graph with the
given subject and predicate IRI, in document order. As with [Graph.Objects], a
repeated statement yields its object once. SubjectsOfType returns every subject in this graph with rdf:type typeIRI, in
document order (deduplicated). Like Graph.SubjectsOfType it scans rather than
keying off the subject index, so it never triggers an index build. Triples iterates the view's statements in document order, projecting each quad
onto its triple. It builds no index and materializes no slice, yielding one
Triple value at a time from the dataset's quads; the only allocation is the
iterator closure itself, a fixed ~56 bytes per call regardless of graph size.
The cost that matters is the pass, not the yield. Triples filters the whole
dataset, so walking it is one full-dataset pass per view — reading N graphs out
of one dataset costs N passes, even for graphs that turn out to be empty. Code
that consumes several graphs at once can beat that by fusing the dispatch into
a single hand-written pass over Dataset.Quads switching on each quad's graph
term. Use Empty to skip a graph without walking it at all: it reads the
dataset's cached counts and never scans.
Per-triple overhead, by contrast, is not worth avoiding: in this package's
Corpus/Grain/SingleGraph Triples benchmarks the iterator beats an equivalent
hand-written single-graph loop at both corpus and per-grain scale, even against
a single-graph dataset whose loop needs no filter at all.
*GraphView : GraphQuery
func (*Dataset).GraphView(graph Term) *GraphView
Kind distinguishes the three RDF term types.
const Blank
const IRI
const Literal
Quad is an RDF statement in a named graph: a triple plus the graph it belongs
to. A zero-value graph term (G) denotes the default graph, so a Quad is a
strict superset of a Triple.GTermOTermPTermSTerm Triple projects the quad onto its statement, dropping the graph term.
func (*Decoder).DecodeQuad() (Quad, error)
func (*Encoder).AppendQuad(b []byte, q Quad) []byte
SyntaxError reports a line that is neither blank, nor a comment, nor a valid
N-Triples/N-Quads statement. Line is 1-based; Text is the offending line,
trimmed and truncated for the message.
It exists so that a truncated document is an error rather than a smaller graph.
A parser that skips what it cannot read turns a half-written dump into a
well-formed, wrong answer, and no caller downstream can tell.
Line counts from the start of the input actually handed to the parser, which is
not the document when a caller feeds it one chunk at a time: a bad line in the
third 1MB chunk of a large dump is reported at its offset within that chunk, and
the number looks entirely plausible. Parse a chunked document through
[NewDecoder] instead -- it reads across an [io.Reader] and numbers lines
continuously -- or add the chunk's line base to Line before showing it to
anyone. // 1-based, relative to the input given to the parser; see the type doc // the offending line, trimmed(*SyntaxError) Error() string
*SyntaxError : error
Canonicalize returns a new Dataset with blank nodes relabeled to their canonical
_:c14nN identifiers and quads in canonical order, leaving d untouched.
LocalName returns the fragment or last path segment of an IRI (the part after
the final '#' or '/').
NewDecoder returns a streaming Decoder reading the given format from r.
For the line-based formats (N-Triples, N-Quads) the decoder reads a line at a
time and holds only one statement at once, so it streams a document of any size
in bounded memory -- with one caveat: a "line" is the bytes up to the next
newline, accumulated without limit, so input that never emits one grows a single
line until it exhausts memory. When r is untrusted, wrap it in an
[io.LimitReader], or a reader that caps bytes-since-newline, before handing it
over. The parser bounds what it holds per statement; it cannot bound what the
input calls one line.
NewIRI, NewBlank and NewLiteral construct the three term kinds.
NewLiteral makes a literal; an empty datatype defaults per RDF (xsd:string, or
rdf:langString when a language tag is present).
ParseJSONLD parses a JSON-LD document into a Graph. It expands compact IRIs and
terms against an inline @context (prefix maps and term definitions, including
"@type":"@id" coercion), and handles @graph, @id, @type, @value/@language/@type
literals, @list, nested node objects, and arrays of any of these. It does not
fetch remote @context documents or run full JSON-LD 1.1 framing.
ParseNQuads parses an N-Quads document into a Dataset, preserving each
statement's graph term; lines carrying only three terms fall in the default
graph. It shares the N-Triples term reader, so the same escaping and leniency
apply — blank, comment and malformed lines are skipped. One private copy of
the input backs every term, so data is free for reuse once it returns.
ParseNQuadsShared is ParseNQuads without the private input copy: terms are
zero-copy views into data itself, saving one input-sized allocation — worth
it when read-heavy consumers parse corpus-scale documents. In exchange the
caller must not modify data while the Dataset, or any Graph or Term derived
from it, remains in use.
ParseNTriples parses an N-Triples document (one "subject predicate object ."
statement per line) into a Graph. It also accepts N-Quads, ignoring any fourth
(graph) term. Blank and comment lines are skipped; a malformed line is a
*SyntaxError naming the line number, so a truncated document does not parse as
a smaller graph. One private copy of the input backs every term, so data is
free for reuse once it returns.
Use [NewDecoder] with [Decoder.SkipMalformed] to tolerate the trailing noise
some real-world dumps carry.
ParseNTriplesShared is ParseNTriples without the private input copy: terms
are zero-copy views into data itself, saving one input-sized allocation. In
exchange the caller must not modify data while the Graph, or any Term drawn
from it, remains in use.
ParseRDFXML parses an RDF/XML document into a Graph. It supports the striped
syntax real bibliographic RDF uses: typed node elements and rdf:Description,
rdf:about / rdf:nodeID / rdf:ID subjects, rdf:resource and rdf:nodeID object
references, nested node elements, literal property values with rdf:datatype and
xml:lang, property attributes, and rdf:parseType="Resource". It does not handle
RDF containers, reification, or rdf:parseType="Literal"/"Collection".
ParseTurtle parses a Turtle document into a Graph. It supports the constructs
real RDF uses: @prefix/@base and SPARQL-style PREFIX/BASE, prefixed names and
IRIs, the `a` keyword, predicate-object lists (`;`) and object lists (`,`),
blank-node labels, `[ … ]` blank-node property lists, `( … )` collections, and
string (including triple-quoted), language-tagged, datatyped, numeric and
boolean literals. Local names are read as the common identifier subset.
TurtleHeader returns the @prefix declaration block for the given prefixes,
terminated by a blank line. Collection writers emit it once.
Package-Level Variables (only one)
ErrCanonComplexity is returned by the canonicalization functions when a graph's
blank-node structure would require more work than maxCanonWork permits — the
guard against the algorithm's exponential worst case on adversarial "poison"
datasets. Well-formed data never approaches the limit.
NQuads is N-Quads; the optional graph label on each line is ignored, so each
statement still yields a triple.
Well-known IRIs.
NTriples is the line-based N-Triples format.
RDFXML is RDF/XML; triples stream as each node element is parsed.
Well-known IRIs.
Turtle streams '.'-terminated statements; @prefix/@base carry forward.
SPARQL-style PREFIX/BASE directives (no '.') are not supported when streaming.
Well-known IRIs.
Well-known IRIs.
The pages are generated with Goldsv0.8.7. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @zigo_101 (reachable from the left QR code) to get the latest news of Golds.
