At scale: huge ontologies and answers you can trust

It's quarter close. Finance asks the agent from Chapter 14 a simple question: "What is Carolyn Inworth's total spend?" Two seconds later: "$502,529.42 across 15 orders." Confident, well formatted, a live query you can read. Finance's spreadsheet says $504,779.42. The difference is the online store, which launched in January, keeps its orders in a new Aurora table, and was wired into Stardog last week by a colleague who typed cust: where the template needed customer:. Nothing failed. No query errored. The graph did exactly what it was told, and the agent reported what came back. Scale that up to a metric fed by thirty sources and you get the author's version of this story: an agent that says $400K when the truth is $500K, with a straight face.

This chapter answers two questions that every team hits right after the demo works. A: at Uber, Amazon, Google or Databricks scale the ontology has thousands of classes and millions of triples, so how does the agent write SPARQL without seeing all of it? B: when an answer needs three sources and one quietly contributes nothing, how do you get "this is supposed to include A, B and C; I couldn't get C; the partial answer is X" instead of a confident wrong number? You'll build the second one yourself: a third source with a deliberately broken mapping, a registry that says what should be there, and agent_v2.py, which catches the miss in code.

16.1Terms in this chapter

16.2Question A: the ontology won't fit in the prompt

Chapter 14's agent cheats in a way that only works at our size. Its system prompt holds a hand-written summary of the whole c360 ontology, 10 classes and 33 properties in about 700 tokens, and get_ontology returns all 48 terms in one go. A real enterprise model is a different animal: thousands of classes across finance, logistics, marketing and HR, tens of thousands of properties, and several near-duplicates of everything ("revenue", "net revenue", "booked revenue"). Even when a context window could technically hold it, you pay for every token on every call, and the model's attention thins out: the right property is somewhere in 200 pages of schema, next to four plausible wrong ones.

The research literature has a name for the hard part: schema linking, matching the words in a question to the right tables and columns (for us, classes and properties). Text-to-SQL researchers treat it as its own step; one 2025 paper calls it "a critical bottleneck in applying existing Text-to-SQL models to real-world, large-scale, multi-database environments." Writing the query once you know the right five terms is the easy part. Finding those five among 50,000 is the job.

Every approach that works at scale does the same thing in some order: narrow the search space, then give the model only the slice it needs. Four techniques, usually combined.

16.3Retrieve a slice: search, then expand one or two hops

Take the words of the question, find the ontology terms whose labels or comments match, then add their neighbors: for a class, the properties whose domain or range it is; for a property, its domain and range classes; plus parents and children. One hop usually suffices; two hops if the question spans a join. The result is an ontology slice: twenty lines instead of the whole model. Press Play:

How you search matters. Lexical search matches word forms. Stardog has it built in: with the database option search.enabled=true, a query can ask its full-text index for literals matching a string. The docs show the predicate form, which binds the matching literal and, optionally, a score:

SELECT ?term ?text WHERE {
  (?text ?score) <tag:stardog:api:property:textMatch> 'price' .
  GRAPH <urn:stardog:training-c360:1.0:model> { ?term rdfs:label ?text }
}

There's also a SERVICE fts:textMatch { … } form (with fts: = tag:stardog:api:search:) that takes extra parameters. Two limits from the docs matter here: search "is not supported over data in virtual sources", which is fine because the ontology is stored, not virtual; and options control which graphs get indexed. That second one bit me when I read your server's settings (read-only): on kit-c360, full-text search is on, but search.index.contexts.filter limits the lexical index to the default graph, so a textMatch over the model graphs will likely come back empty. Changing that means reindexing, and it's not worth it at our size, so agent_v2.py defaults to a plain CONTAINS scan over labels and comments. That's instant for a few hundred terms and too slow for millions, which is where the index earns its keep.

Lexical search also has a vocabulary problem. Search the c360 labels for "spend" and you get exactly one hit: :Big_Spender, Chapter 10's derived class for customers with any order over $250. That's the wrong concept for "total spend", and the right one, :purchasePrice, is labeled "price". Semantic search compares vector embeddings instead of word forms, so "spend" can land near "price". Stardog's docs describe a semantic mode in the same full-text machinery (search.semantic.enabled, queried with fts:semantic true, using a bundled embedding model), and your server has it switched on for the two model graphs; the same setting lists Voicebox's question graph, so this is presumably the index Voicebox uses to match a question's concepts to the ontology (Chapter 13). The third fix is cheaper than either: put the business vocabulary in the graph. Once Section 16.8 adds a metric labeled "total spend", plain CONTAINS finds it, and it points at :Order.

16.4Route first: modular ontologies

Before searching everything, pick the business domain. Large ontologies are usually built as modules (one named graph, or one file, per domain: sales, logistics, HR), so a cheap first step, "which domain is this question about?", shrinks the search space by ten or a hundred times. Stardog already gives you the unit: Chapter 10's reasoning schemas are named sets of model graphs, and a query uses exactly one.

Uber has described this pattern publicly for its internal text-to-SQL tool, QueryGPT (Uber engineering blog, September 2024). An intent agent maps the question "to one or more business domains/workspaces", curated collections of tables and SQL samples for areas such as Ads or Mobility. A table agent then proposes tables, which the user can accept or edit, and a column prune agent removes irrelevant columns before the SQL is written. Intent, then workspace, then tables, then columns: each step narrows the next. That's routing plus schema linking, done in stages.

16.5Let the agent explore, like you would

The fourth technique hands the search to the model. Instead of stuffing a slice into the prompt, give it tools to explore the schema step by step, the way you'd poke at information_schema in a new job:

• search_ontology(term): terms whose label or comment mentions a word, each with its one-hop neighbors.
• describe_class(iri): one class or property and its neighborhood, the graph's \d orders.

A typical run: the question says "total spend"; the model calls search_ontology("spend"), sees the metric and :Big_Spender, reads the comments, picks the metric, calls describe_class(":Order") to see its properties, and only then writes SPARQL. Three cheap calls on stored triples; neither warehouse wakes up. Here are the two search functions from agent_v2.py:

def _hits(term: str) -> list:
    graphs = " ".join(f"<{g}>" for g in MODEL_GRAPHS + (REGISTRY,))
    if SEARCH_MODE == "lexical":     # Stardog full-text index (search.enabled=true)
        match = f"(?text ?score) <tag:stardog:api:property:textMatch> {literal(term)} ."
    elif SEARCH_MODE == "semantic":  # Stardog semantic search (search.semantic.enabled=true)
        match = f"SERVICE fts:textMatch {{ [] fts:query {literal(term)} ; fts:semantic true ; fts:result ?text }}"
    else:                            # plain scan: fine for thousands of labels, not millions
        match = f"FILTER(CONTAINS(LCASE(STR(?text)), LCASE({literal(term)})))"
    q = f"""SELECT DISTINCT ?term WHERE {{
  {match}
  VALUES ?g {{ {graphs} }}
  GRAPH ?g {{ ?term rdfs:label|rdfs:comment|ops:definition ?text }}

The hits then go to _slice(), one query that adds each term's neighbors through so:domainIncludes, so:rangeIncludes, rdfs:subClassOf and ops:requires, and formats them like get_ontology does. The tool itself:

def search_ontology(term: str) -> str:
    """Tool 3. Find ontology terms whose label/comment mentions `term`, plus 1-hop neighbors."""
    try:
        hits = _hits(term)
        if not hits:
            return f'No ontology term mentions "{term}". Try a synonym (spend -> price, buyer -> customer).'
        return _slice(hits)
    except (ValueError, RuntimeError, requests.RequestException) as e:
        return str(e)

Two details. literal() strips everything but letters, digits and a few punctuation marks before a word goes into a query, and iri() refuses anything that isn't a well-formed IRI, so neither can smuggle SPARQL in. And every query still goes through Chapter 14's guard(): read-only, capped LIMIT.

16.6Verified examples, retrieved on demand

Chapter 13 introduced few-shot examples: a handful of solved question-and-query pairs placed in the prompt so the model copies their shape. At scale you can't include all of them either, so you retrieve the few most similar to the current question, and you only keep examples a person has checked. This is the most consistent pattern across vendors, each in its own words:

Notice the shared idea: a human-verified query is an asset, not a hint. Genie and Cortex Analyst even tell the user when an answer came from one. Your five competency questions from Chapter 14, with their SQL-checked answers, are the start of such a library, and in Section 16.12 they become golden tests.

16.7Question B: the answer that's quietly missing a source

Now the harder question. Schema retrieval makes the agent pick the right terms. It can't make the data behind them complete. Look at how our stack fails:

• A wrong IRI template (Chapter 9): two graphs name the same customer differently, the join matches nothing, "no error, no rows".
• A graph you may not read (Chapter 12): silently dropped from the query's dataset. Empty, not an error.
• A source nobody mapped: the rows exist in a table, but no virtual graph mentions it. The graph can't miss what it was never told about.
• The open world (Chapter 5): absent triples are unknown, not false, so an empty result is a perfectly valid answer.

Every one of these produces a smaller number, not an error. And the LLM can't catch it, because it only sees what came back: 15 orders summing to $502,529.42 look exactly like a complete answer. Telling the model "be careful about missing data" in the prompt doesn't help; it has nothing to be careful with. What's needed is a completeness check: code that compares the sources a metric should include with the sources that actually contributed, and says which are missing and why. So the rule of this half of the chapter:

Let's make the failure real first.

16.8The source registry: what should be there

You can't notice a missing source unless something says it should be present. That something is a source registry: a few triples, in their own named graph <urn:c360:registry>, saying which sources provide which classes, how they join, and which classes each metric requires. It's data, so it lives in the graph next to the ontology, readable by SPARQL, versioned like everything else. We need a tiny vocabulary for it, ops: = tag:c360:ops:, with five terms:

Notice what the registry does not say: it doesn't copy the mapping. It states the contract ("online orders join on customer:{cid}"), and a probe checks reality against it. The provides half could even be generated: the catalog database that came up during the kit cleanup stores Stardog's mappings as R2RML triples maps, with their IRI templates (such as urn:stardog:demos:c360:customer:{cid}) linked to tables and columns (whether and when it picks up your virtual graphs depends on Stardog's catalog jobs, so check before relying on it). But the requires half, "total spend must include online orders", is a business decision that no catalog can infer. Someone has to write it down once.

16.9Probes: health, key alignment, per-source subtotals

With the expected list in hand, get_total_spend runs three small checks per source, one query at a time so that one broken source can't hide the others:

1. Health and permission probe. SELECT ?s WHERE { GRAPH <g> { ?s a :Order } } LIMIT 1. An error means unreachable. A row means data of the right class exists. Nothing at all is ambiguous: empty, unreadable or undefined all look the same through GRAPH. So the probe asks again with SERVICE <g>, which (per Chapter 12 and the Virtual Graph Security docs) raises an error where GRAPH stays silent, and classifies the error text. Triples of some other class mean the mapping emits the wrong class.
2. Key-overlap probe (key alignment). Sample up to 100 distinct customer keys from the source and count how many are customers in the master graph. The share is the source's key overlap. Redshift: 100%. Online orders with the bad template: 0%. A healthy source can have a few orphans; 0% means the IRI template doesn't match, and the probe also tests the sample against the registry's template to say what the keys look like instead.
3. Per-source subtotal. SUM(?p * ?q) and COUNT for the customer, one source at a time, so the answer can say where every dollar came from.

Press Play to watch the check catch the miss that Step 3 didn't:

Here is every failure mode, how the code tells them apart, and what the answer then says:

Two honest caveats. The error classification reads Stardog's error text for words like "permission" or "not found"; the exact messages vary between versions, so an unrecognized error falls back to "unreachable" with the raw text attached, which is still a correct "missing". And the last row has a mirror image nothing can catch: a source that exists, is unmapped, and isn't in the registry. That's a governance gap, not a query problem; the fix is a process (no new source goes live without a registry entry), the same way no new table ships without a data contract (Section 16.13).

16.10The answer contract

The last piece stops the LLM from rounding a partial answer into a complete-sounding sentence. get_total_spend returns an answer contract, a fixed JSON shape: {value, complete, expected, included, missing: [{source, reason}]}, plus the per-source subtotals and a ready-made summary sentence built by code. This is what it returns for Carolyn after Step 4 (the sample key's number may differ):

{
  "metric": "ops:total_spend",
  "value": "502529.42",
  "complete": false,
  "expected": ["aurora_online_c360", "redshift_c360"],
  "included": ["redshift_c360"],
  "missing": [{"source": "aurora_online_c360",
               "reason": "mapping mismatch (0% customer-key overlap in 100 sampled keys; keys look like urn:stardog:demos:c360:cust:5, registry expects urn:stardog:demos:c360:customer:{cid})"}],
  "subtotals": {"redshift_c360": {"spend": "502529.42", "orders": 15, "key_overlap": 1.0}},
  "customers": ["Carolyn Inworth"],
  "warnings": [],
  "summary": "Partial: $502,529.42. Expected sources aurora_online_c360, redshift_c360; aurora_online_c360 missing because mapping mismatch (0% customer-key overlap in 100 sampled keys; keys look like urn:stardog:demos:c360:cust:5, registry expects urn:stardog:demos:c360:customer:{cid})."
}

complete is computed, never judged: true only when there's a value, the registry listed at least one source, and nothing is missing. The system prompt then makes the presentation non-negotiable. Here's what agent_v2.py appends to Chapter 14's prompt:

SYSTEM_PROMPT = v1.SYSTEM_PROMPT + """

Chapter 16 additions:
- Total spend (for a customer by name or cid, or for a state) MUST come from get_total_spend,
  never from your own run_sparql query: only get_total_spend checks that every source that
  should feed the number actually did.
- get_total_spend returns an answer contract (JSON). Begin your answer with its "summary"
  field, copied verbatim. If "complete" is false, never call the value the total: say it is
  partial, and list every entry in "missing" with its reason. Do not guess what is missing.
- If the question uses a word you can't map to the ontology above, call search_ontology with
  that word (or describe_class with a term) before writing SPARQL."""

Why a pre-built summary instead of letting the model phrase it? Because paraphrase is where "partial" turns into "approximately" and then into nothing. Copying a sentence verbatim is the one thing LLMs do very reliably, and the golden tests in Section 16.12 check that it happened.

16.11The code: agent_v2.py

agent_v2.py (about 380 lines) imports Chapter 14's agent.py and keeps run_sparql, get_ontology, guard(), the environment variables and the OpenRouter switch unchanged. It adds the three schema tools from Part A, the completeness check, a longer prompt and a --spend mode that runs the check without any LLM. The heart of it:

def probe(src: dict) -> tuple:
    """Health probe. Returns (status, detail). status 'ok' means rows of the class exist."""
    g, cls = iri(src["graph"]), iri(src["cls"])
    try:
        if select(f"SELECT ?s WHERE {{ GRAPH {g} {{ ?s a {cls} }} }} LIMIT 1"):
            return "ok", ""
        if select(f"SELECT ?s WHERE {{ GRAPH {g} {{ ?s ?p ?o }} }} LIMIT 1"):
            return "mapping_mismatch", f"the graph has triples but no {short(src['cls'])} (wrong class in the mapping?)"
    except (RuntimeError, requests.RequestException) as e:
        return classify_error(str(e)), str(e)[:200]
    # Empty through GRAPH: truly empty, unreadable (silently dropped, Chapter 12) or not defined.
    # SERVICE fails loudly where GRAPH stays silent, so ask again that way.
    try:
        rows = select(f"SELECT ?s WHERE {{ SERVICE {g} {{ ?s ?p ?o }} }} LIMIT 1")
    except (RuntimeError, requests.RequestException) as e:
        return classify_error(str(e)), str(e)[:200]
    if rows:
        return "no_permission", "visible through SERVICE but not through GRAPH"
    return "no_data", "reachable, but the source returned no rows"


def key_overlap(src: dict, master: dict) -> tuple:
    """Share of the source's customer keys that are known customers. Returns (share, n, example)."""
    keys = [r["k"] for r in select(f"""SELECT DISTINCT ?k WHERE {{ GRAPH {iri(src['graph'])} {{
  ?o a {iri(src['cls'])} ; {iri(src['keyProp'])} ?k }} }} LIMIT {KEY_SAMPLE}""")]
    keys = [k for k in keys if SAFE_IRI.match(k)]
    if not keys:
        return 0.0, 0, ""
    values = " ".join(f"<{k}>" for k in keys)
    known = select(f"""SELECT (COUNT(DISTINCT ?k) AS ?n) WHERE {{ VALUES ?k {{ {values} }}
  GRAPH {iri(master['graph'])} {{ ?k a :Customer }} }}""")
    n = int(known[0]["n"]) if known else 0
    return n / len(keys), len(keys), keys[0]

def get_total_spend(customer_name=None, cid=None, state=None) -> dict:
    """Tool 5. Total spend with a deterministic completeness check. Returns the answer contract."""
    sources, master = read_registry()
    expected = [name_of(s["graph"]) for s in sources]
    if not sources or not master:
        return contract(None, expected, [], [{"source": "registry", "reason":
                        f"no sources or customer master declared in <{REGISTRY}>"}], {}, [], [])
    people = find_customers(master, customer_name, cid, state)
    if not people:
        return contract(None, expected, [], [], {}, [], [],
                        note=f"no matching customer in {name_of(master['graph'])}")
    cust = [p["c"] for p in people]
    total, included, missing, subtotals, warnings = Decimal("0"), [], [], {}, []
    for src in sources:
        name = name_of(src["graph"])
        status, detail = probe(src)
        if status != "ok":
            missing.append({"source": name, "reason": status.replace("_", " ") + (f" ({detail})" if detail else "")})
            continue
        try:
            share, n, example = key_overlap(src, master)
            if share == 0:
                hint = "" if template_regex(src["template"]).match(example) else \
                       f"; keys look like {example}, registry expects {src['template']}"
                missing.append({"source": name, "reason":
                                f"mapping mismatch (0% customer-key overlap in {n} sampled keys{hint})"})
                continue
            if share < OVERLAP_MIN:
                warnings.append(f"{name}: only {share:.0%} of {n} sampled customer keys are known customers")
            amount, orders = subtotal(src, cust)
        except (ValueError, RuntimeError, requests.RequestException) as e:
            missing.append({"source": name, "reason": f"{classify_error(str(e))} ({str(e)[:200]})"})
            continue
        total += amount
        included.append(name)
        subtotals[name] = {"spend": f"{amount:.2f}", "orders": orders, "key_overlap": round(share, 3)}
    return contract(total, expected, included, missing, subtotals,
                    [p["name"] for p in people][:10], warnings)

A reading guide:

• read_registry() (not shown) asks <urn:c360:registry> which graphs provide what ops:total_spend requires, and which graph is the customer master. No registry means complete: false, never a silent pass.
• probe() is Section 16.9's health and permission probe, including the SERVICE retry.
• key_overlap() samples keys (Chapter 14's guard caps it at 100) and counts known customers with one VALUES query against the master graph.
• get_total_spend() skips any source that fails a probe, records why, and adds up only what passed. The value is always the sum of what was actually included, so a partial answer is still a correct partial answer.

I tested the completeness logic offline against a fake Stardog covering every row of the failure table, a missing registry, an unknown customer and an injection attempt in the customer name; every query it generates also parses as standard SPARQL 1.1. It hasn't been run against your server yet, which is what the next steps do.

16.12Golden tests: eval.py

A check you only run by hand stops being run. eval.py turns the competency questions into golden tests: each question has an answer verified in SQL, the harness asks the agent, extracts the numbers, and compares within a cent. Spend questions carry an extra completeness assertion: the number must come from get_total_spend, and an incomplete contract fails the test even if the model admits it. Here it is in full:

#!/usr/bin/env python3
"""eval.py - golden tests for the c360 agent (Chapter 16).

Each case is a competency question with an answer verified in plain SQL (DBeaver).
The harness asks agent_v2, pulls the numbers out of the answer, compares them within a
tolerance, and checks completeness: a spend answer must come from get_total_spend, and
if that contract says complete=false, the answer must say "Partial".

Usage:  python eval.py            (all cases; each one calls the LLM and Stardog)
        python eval.py --direct   (spend cases only, straight through get_total_spend, no LLM)
Exit code 0 when every case passes, so you can run it in CI.
"""
import json, re, sys
import agent_v2

CASES = [  # expected values: SQL over Aurora + Redshift, after Chapter 16 step 6
    {"q": "What is Carolyn Inworth's total spend?", "number": 504779.42,
     "spend": {"customer_name": "Carolyn Inworth"}},
    {"q": "What is the total spend of all customers in Wisconsin?", "number": 839445.23,
     "spend": {"state": "Wisconsin"}},
    {"q": "How many customers live in Texas?", "number": 53},
    {"q": "How many purchases are recorded in Redshift?", "number": 2500},
    {"q": "Who is the top spender in Wisconsin?", "text": "Carolyn Inworth"},
]
TOLERANCE = 0.01


def numbers(text: str) -> list:
    return [float(n.replace(",", "")) for n in re.findall(r"\d[\d,]*(?:\.\d+)?", text)]


def check_answer(case: dict, answer: str, log: list) -> list:
    """Return a list of failure reasons (empty = PASS)."""
    fails = []
    if "number" in case and not any(abs(n - case["number"]) <= TOLERANCE for n in numbers(answer)):
        fails.append(f"expected {case['number']:,} in the answer")
    if "text" in case and case["text"].lower() not in answer.lower():
        fails.append(f"expected '{case['text']}' in the answer")
    if "spend" in case:  # completeness assertion
        contracts = [json.loads(r) for name, _, r in log if name == "get_total_spend"]
        if not contracts:
            fails.append("spend was not computed by get_total_spend (no completeness check)")
        elif not contracts[-1].get("complete"):
            if "partial" not in answer.lower():
                fails.append("contract says incomplete but the answer does not say 'Partial'")
            fails.append("incomplete: " + contracts[-1].get("summary", ""))
    return fails


def run(direct: bool) -> int:
    failed = 0
    for case in CASES:
        if direct:
            if "spend" not in case:
                continue
            c = agent_v2.get_total_spend(**case["spend"])
            answer, log = c["summary"], [("get_total_spend", case["spend"], json.dumps(c))]
        else:
            log = []
            answer = agent_v2.ask(case["q"], log)
        fails = check_answer(case, answer, log)
        failed += bool(fails)
        print(f"{'FAIL' if fails else 'PASS'}  {case['q']}")
        for f in fails:
            print(f"      - {f}")
        if fails:
            print(f"      answer: {answer[:300]}")
    print(f"\n{failed} failed" if failed else "\nall passed")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(run("--direct" in sys.argv))

The expected values were computed from the same rows you loaded (the kit's CSVs) plus Step 1's formula, and they reproduce Chapter 9's verified numbers exactly. Run python eval.py --direct before Step 6 and both spend cases fail, with the contract's partial summary printed; after the fix, both pass. python eval.py runs all five through the LLM. Wire it into CI and a future mapping change that drops a source fails a build instead of a board meeting. (After the cleanup below, the two spend numbers go back to $502,529.42 and $837,195.23.)

16.13Where this sits in industry practice

None of this is new to data engineering; it's the same discipline applied one layer up, where an agent reads the data. Four ideas you'll meet by name:

• Data contract. An agreement between the team that produces a dataset and the teams that consume it: schema, meaning, freshness, and who is told when it changes. Our registry entry "online orders provide :Order and join on customer:{cid}" is a two-line data contract, and the key-overlap probe enforces it.
• Data observability. Continuously checking data the way you monitor services. Monte Carlo popularized "five pillars" (freshness, distribution, volume, schema, lineage), and Great Expectations lets you write assertions such as expect_table_row_count_to_be_between or expect_column_values_to_not_be_null. Our probes are volume and schema checks run at question time.
• Lineage and provenance. Recording where a value came from. The W3C standard vocabulary is PROV-O (a W3C Recommendation since April 2013): prov:Entity, prov:Activity, prov:Agent, and links such as prov:used, prov:wasGeneratedBy and prov:wasDerivedFrom. Stardog's own catalog database uses prov:used and prov:generated to record its materialization runs. The contract's included and subtotals are provenance in miniature; a next step would emit them as PROV-O triples.
• Metrics layer. Define a metric once, with its inputs, and have every tool use that definition: the dbt Semantic Layer (powered by MetricFlow, metrics in YAML on semantic models), Cube (measures and dimensions defined once, served over SQL, REST and GraphQL), and Looker's LookML (measures in views). ops:total_spend ops:requires :Order is the same idea in triples, and it's what lets code, rather than a model, know what "complete" means.