SHACL: when you do want rules to reject data

Try an experiment in your head. You INSERT a customer into Postgres with two email addresses, one of them "not-an-email", and an address id that doesn't exist. Postgres won't have it: a single email column can't hold two values, a CHECK rejects the malformed one, and the foreign key rejects the unknown address. Insert the same customer into Stardog as triples and it says "OK," then and later, reasoning or not. Which brings us to your question: what exactly does SHACL mean, and why do people use it? SHACL stands for Shapes Constraint Language, a W3C standard (a Recommendation since July 2017) for writing down what acceptable data looks like and checking real data against it. RDFS and OWL are designed never to reject anything. SHACL is the part of the RDF world that says "no, that's wrong," and tells you which node, which property and which rule.

11.1Why the ontology never said no

Two ideas from earlier chapters explain it. The open-world assumption (Chapter 5): a missing fact is unknown, not false, so a customer with no :email triple is not an error, where a closed-world database would reject the NULL. Axioms infer rather than reject (Chapter 6): if the model said :address rdfs:range :Address and an :address pointed at a product, RDFS would infer the product is an Address, which is why the kit uses the documentation-only so:rangeIncludes. Neither ever checks anything.

A tiny example of the gap. These two triples are fine to OWL:

<urn:stardog:demos:c360:customer:9999> :email "bad.customer@example.com" , "not-an-email" .

Two emails? A customer can have several. A malformed one? OWL only sees a string. If you come from Postgres, you want email varchar NOT NULL UNIQUE CHECK (email ~ '@'). SHACL is how you say that about a graph.

11.2SHACL's vocabulary, broken down

SHACL is itself written in RDF (Turtle), using the namespace sh: = http://www.w3.org/ns/shacl#.

The smallest complete shape, "every Customer must have an email":

:EmailRequired a sh:NodeShape ;
  sh:targetClass :Customer ;
  sh:property [ sh:path :email ; sh:minCount 1 ] .

Against the data from Section 11.1, customer:9999 passes (it has at least one email). Add sh:maxCount 1 and it fails with "more than 1 value." Add sh:pattern "^[^@ ]+@[^@ ]+[.][^@ ]+$" and "not-an-email" fails too. That's the entire idea; the rest is detail.

11.3SHACL for a database person

The translation table to keep open:

11.4The CustomerShape, line by line

The exercise's shape: every Customer has exactly one well-formed email and at least one address that is an Address:

:CustomerShape a sh:NodeShape ;
  sh:targetClass :Customer ;
  sh:property [
    sh:path :email ;
    sh:minCount 1 ;
    sh:maxCount 1 ;
    sh:datatype xsd:string ;
    sh:pattern "^[^@ ]+@[^@ ]+[.][^@ ]+$" ;
    sh:message "A customer needs exactly one well-formed email."
  ] ;
  sh:property [
    sh:path :address ;
    sh:minCount 1 ;
    sh:class :Address ;
    sh:message "A customer's address must be an Address."
  ] .

Top to bottom: a node shape that targets every :Customer (in Stardog including subclass instances, because shacl.targetClass.simple is false). Each sh:property [ … ] is a property shape, an unnamed (blank) node holding the checks for one path. minCount 1 ; maxCount 1 means exactly one value: NOT NULL plus single-valued. A plain "…" literal is an xsd:string, so sh:datatype passes for normal text. The regex uses [.] instead of \. to avoid escaping inside Turtle strings. sh:class :Address requires the value to have rdf:type :Address in the data being validated: the foreign-key check. sh:message is optional text copied into each result.

11.5Running validation in Stardog 12

Stardog calls its SHACL engine ICV (Integrity Constraint Validation), a name you'll see in options, the CLI and the HTTP API. Four ways to run it, all read-only for your data:

The VALIDATE grammar from the docs:

VALIDATE (ALL | [<IRI>+] [GRAPH <IRI>+])
  [USING SHAPES (<IRI>+ | GRAPH <IRI>+ | <QuadData>)]
  [LIMIT <int>] [LIMIT PER SHAPE <int>]

In words: say what data to check (all, specific nodes, or named graphs) and which shapes to use (shape IRIs, shapes stored in named graphs, or shapes inline in curly braces). LIMIT caps the number of results.

Where shapes live, and why you name them. Shapes are ordinary triples, stored in a named graph like anything else. In the 11.7 exercise you'll put yours in a scratch graph, <urn:book:ch11:shapes>. The question is which shapes a validation uses. There are two ways to run it:

It works like dbt test, which runs every test file it finds in the project. Forget a scratch test file and it runs too, every time. Same here: if you forget to drop <urn:book:ch11:shapes>, next month's VALIDATE ALL still checks every customer against this chapter's practice :CustomerShape, and reports violations nobody asked for. So the 11.7 exercise always names its shapes graph with USING SHAPES GRAPH, and drops the scratch graphs at the end.

11.6Where SHACL fits in a federated setup

In our architecture the data isn't in Stardog; it's SQL rows seen through mappings. What could go wrong, and where?

• Upstream data problems: a NULL email in Aurora, or a cid in Redshift with no matching customer. SHACL can find these but not fix or block them, because nothing is written through Stardog; the fix happens in Aurora or Redshift. What the report adds is the wording. Say customer 225's email is NULL. SQL describes that as "row cid = 225 of c360.customer has NULL in email". The mapping then produces no :email triple, sh:minCount 1 fails, and the report says "customer:225 has no :email": the same problem, in the names the business and the agent use.
• Mapping mistakes, where SHACL shines for us: a wrong IRI template (Chapter 9's buyer-vs-customer bug: if the Redshift mapping minted buyer:{cid}, every order would say :purchasedBy buyer:225, the federated join would quietly find nothing, and a shape saying :purchasedBy must point at a :Customer would fail for every order), a missing cast (Chapter 9's xsd:float money becomes a sh:datatype xsd:decimal violation), a column mapped to the wrong property. The SQL looks fine, the triples are subtly wrong, and a shape catches them.

Can Stardog validate a virtual graph directly?

The main documentation page, Data Quality Constraints (docs.stardog.com/data-quality-constraints), doesn't mention virtual graphs. On the Stardog Community forum, Stardog staff have answered twice:

• "SHACL validation for virtual graphs" (community.stardog.com/t/3238): Pavel Klinov explains that Stardog can't validate data before it enters a virtual graph (writes go straight to the source) and suggests keeping the shapes in Stardog and running stardog icv report --named-graph virtual://graph1 <db>. For big data he suggests a cache.
• "SHACL validation on a Virtual graph" (community.stardog.com/t/4912): Paul Jackson gives the VALIDATE form, which needs a query hint: #pragma simple.target on, then VALIDATE GRAPH <virtual://graph_name> USING SHAPES { … } (lowercase virtual://). The user reported still getting sh:conforms true for shapes that should fail; the thread shows no resolution.

So it's supported in principle, not officially documented, and at least one user saw false "all clear" results. We'll do the documented thing: validate a small materialized sample in the scratch graph <urn:book:ch11>. An optional read-only stretch at the end tries the virtual graph with a built-in lie detector.

11.7Try it: validate a sample, break it, fix it

Two scratch graphs, deleted at the end: <urn:book:ch11> for the data and <urn:book:ch11:shapes> for the shapes. The data uses real values for Jolynn (customer 1) and Carolyn (customer 225), plus a deliberately bad customer 9999, who doesn't exist in Aurora.

11.8Reading a validation report

The kind of report step 3 returns, trimmed to one result:

[ a sh:ValidationReport ;
  sh:conforms false ;
  sh:result [
    a sh:ValidationResult ;
    sh:resultSeverity sh:Violation ;
    sh:sourceShape _:b0 ;                                  # the email property shape
    sh:sourceConstraintComponent sh:PatternConstraintComponent ;
    sh:focusNode <urn:stardog:demos:c360:customer:9999> ;
    sh:resultPath :email ;
    sh:value "not-an-email" ;
    sh:resultMessage "A customer needs exactly one well-formed email."
  ] ] .

Read it like a failed test: sh:focusNode is the bad row's primary key, sh:resultPath is the column, sh:value is the bad cell (when there is one), sh:sourceConstraintComponent says which check failed, and sh:resultSeverity says how much to care.

sh:conforms is the verdict, not a setting. Every report starts with it, and what follows depends on it:

[ a sh:ValidationReport ;
  sh:conforms false ;
  sh:result [ … pattern, customer:9999 … ] ;
  sh:result [ … maxCount, customer:9999 … ] ;
  sh:result [ … class, customer:9999 … ] ] .

[ a sh:ValidationReport ;
  sh:conforms true ] .

false comes with one sh:result per problem; true comes alone. Nodes that pass, like Jolynn and Carolyn, never appear in either. There is no setting that makes it true: you fix the data and run VALIDATE again.

Optional stretch (read-only): try the virtual graph, with a lie detector

Curious whether direct virtual-graph validation works on our server? VALIDATE only reads, so this is safe. The trick is a must-fail check: a check you know for certain should fail. If it passes anyway, the checking itself is broken, and you can't trust its "all clear". Ours is a shape that must fail: aurora_c360_safe has no :ssn (Chapter 8), so "every customer has an ssn" must fail. If it reports conforms true, validation isn't really seeing the virtual data, so ignore any "all clear."

#pragma simple.target on
PREFIX :   <tag:stardog:api:ecomm:>
PREFIX sh: <http://www.w3.org/ns/shacl#>
VALIDATE GRAPH <virtual://aurora_c360_safe>
USING SHAPES {
  :MustFailShape a sh:NodeShape ;
    sh:targetClass :Customer ;
    sh:property [ sh:path :ssn ; sh:minCount 1 ] .
}
LIMIT 5

Expected if it works: conforms false with 5 results (capped by LIMIT 5; the true number is 500). Aurora may take about 20 seconds to wake. If that works, swap the must-fail shape for the two sh:property blocks of :CustomerShape and rerun. My read-only SQL check found all 500 real customers have exactly one well-formed email and an existing address row, so a working validator should report conforms true. If Studio rejects the syntax or the must-fail shape passes, note it: that's useful information, not a failure on your part.