Skip to content

Bulk Document Ingest

index_documents() turns row-shaped data — a HuggingFace dataset, a DataFrame's to_dict("records"), a JSONL file — into nodes, relationships, and embeddings in one call.

rows = [
    {"id": "1", "text": "Graph databases store nodes and edges", "year": 2024},
    {"id": "2", "text": "Vector search finds nearest neighbours", "year": 2025},
]

db.create_vector_index("default", dim=384, embedding_function=embedder)
report = db.index_documents(rows, label="Document")

print(report)  # IndexReport(2 created, 0 updated, 0 relationships, 2 embedded)

Every key other than the relationships field becomes a node property, so year above is queryable immediately. Set copy_attributes=False to keep only id and text, or pass properties=["id", "year"] for an explicit allowlist.

Relationships

Point relationships_key at the field holding each row's outgoing edges. Both shapes work:

rows = [
    {"id": "1", "text": "...", "links": ["2", "3"]},
    {"id": "2", "text": "...", "links": [
        {"id": "3", "type": "CITES", "properties": {"weight": 2}},
    ]},
    {"id": "3", "text": "...", "links": []},
]

report = db.index_documents(rows, label="Paper", relationships_key="links")

A bare id uses default_rel_type (RELATED_TO); the mapping form takes type and properties.

Nodes are created in a first pass and relationships in a second, so a row may reference a document that appears later in rows — forward references resolve. Targets that name an id no row supplied are skipped and collected:

if report.unresolved:
    print(f"{len(report.unresolved)} dangling references: {report.unresolved[:5]}")

A dataset slice legitimately references rows outside the slice, so this is not an error. But a long unresolved list usually means id_key does not match the ids used in the relationship field.

Identity and Re-ingest

id_key (default "id") gives rows an external identity. When upsert=True (the default), re-ingesting a row with a known id updates that node instead of creating a duplicate:

db.index_documents(rows, label="Paper")           # 3 created
db.index_documents(updated_rows, label="Paper")   # 0 created, 3 updated

Pass upsert=False to always create, or id_key=None for data with no identity — note that relationships cannot be resolved without an id key.

report.ids maps external ids to node ids, which is how you connect this data to the rest of the graph afterwards:

report = db.index_documents(rows, label="Paper")
db.create_relationship(report.ids["1"], author_node.id, "AUTHORED_BY")

Embedding

Text is embedded in batches using the vector index's embedding function:

db.index_documents(rows, label="Paper", index="papers_vec", batch_size=512)
  • batch_size controls how many texts go to the embedder per call. Larger batches are faster for API-backed embedders and use more memory for local ones.
  • Rows with no text_key, or with blank text, are stored as nodes but not embedded.
  • index=None skips embedding entirely — useful for a structure-only load.

Full-Text

Pass configure_fts=True to register a full-text index over label/text_key at the same time, so the documents are reachable by both retrieval paths:

db.index_documents(rows, label="Paper", configure_fts=True)

db.semantic_search("nearest neighbours", k=10)
db.text_search("nearest neighbours", k=10)

It is off by default because it creates an index you did not ask for. If the ingest fails, an index registered by this call is removed again — one you had already created is left alone.

You can also call create_text_index() yourself, but do it before loading: it registers the configuration and indexes rows as they are written. Documents already in the database need an explicit rebuild to become searchable:

db.create_text_index("node", "Paper", ["text"])   # before
db.index_documents(rows, label="Paper")

# or, if the documents are already loaded
db.create_text_index("node", "Paper", ["text"])
db.rebuild_text_index()

The ingest is not atomic

Rows are written as they are read, so a failure part-way through — a bad row, an embedding provider timing out — leaves everything before it in the database. That is deliberate: the alternative is holding a write transaction open across every embedding call, which on a hosted embedder blocks writers for minutes.

To resume, re-run with the remaining rows. With upsert=True (the default) re-feeding rows that already landed updates them instead of duplicating, so replaying the whole batch is safe. If you need all-or-nothing and your embedder is local and fast, wrap the call in begin_transaction()/rollback() yourself.

End to End

db = GrafitoDatabase("papers.db")
db.create_vector_index("default", dim=384, embedding_function=embedder)

report = db.index_documents(
    dataset,                      # any iterable of dicts
    label="Paper",
    text_key="abstract",
    relationships_key="citations",
    default_rel_type="CITES",
    batch_size=512,
)
db.create_text_index("node", "Paper", ["abstract"])

# Retrieve as a graph, then rank within the result
sub = db.semantic_subgraph("retrieval augmented generation", k=50, expand=1)
for hit in db.centrality("pagerank", graph=sub.to_networkx(), limit=10):
    print(hit["node"].properties["title"], hit["score"])

API Reference

grafito.ingest_report.IndexReport dataclass

What :meth:~grafito.GrafitoDatabase.index_documents actually did.

unresolved is the part worth checking: relationship targets that named an id no document supplied. They are skipped rather than raising, because a dataset slice legitimately references rows outside the slice — but a long list here usually means id_key does not match the ids used in the relationship field.

Source code in grafito/ingest_report.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@dataclass
class IndexReport:
    """What :meth:`~grafito.GrafitoDatabase.index_documents` actually did.

    ``unresolved`` is the part worth checking: relationship targets that named
    an id no document supplied. They are skipped rather than raising, because a
    dataset slice legitimately references rows outside the slice — but a long
    list here usually means ``id_key`` does not match the ids used in the
    relationship field.
    """

    nodes_created: int = 0
    nodes_updated: int = 0
    relationships_created: int = 0
    #: Number of nodes whose text was embedded.
    embedded: int = 0
    #: External id -> node id, for every row that carried an id.
    ids: dict[Any, int] = field(default_factory=dict)
    #: Relationship targets that could not be resolved to a node.
    unresolved: list[Any] = field(default_factory=list)

    @property
    def nodes(self) -> int:
        """Total nodes touched, created or updated."""
        return self.nodes_created + self.nodes_updated

    def __str__(self) -> str:
        parts = [
            f"{self.nodes_created} created",
            f"{self.nodes_updated} updated",
            f"{self.relationships_created} relationships",
            f"{self.embedded} embedded",
        ]
        if self.unresolved:
            parts.append(f"{len(self.unresolved)} unresolved")
        return f"IndexReport({', '.join(parts)})"

nodes property

Total nodes touched, created or updated.