Verifying a control framework against the regulation¶

A control framework is a set of claims about a law. Almost nobody checks them. This is a note on regcheck, a small tool that does — and on why building it turned out to be the only way to make the EU AI Act control work trustworthy, whether the work is done by a person or by an AI agent.

It is written to be reusable. The tool is regulation-agnostic, and the working method around it is the part that transfers: it is a way of planning agent work against a legal text so that the output can be checked by someone who was not there.

Status

Working notes, written while the AI Act control set is being migrated. The tool is real and runs; the control set is partial. Figures below are as at 6 August 2026 and will move.

The post that goes with this page: A control framework is six hundred unchecked claims about a law.

The problem¶

A control framework says things like: "Article 10(2)(c) requires you to document data-preparation operations." That is a claim with three parts — the provision exists, it says roughly that, and nothing has repealed it. A framework of 200 controls makes about 600 such claims.

Nobody checks them. Not because people are careless, but because there is no mechanism. You can review a control framework the way you review a document: read it, form a view, note some corrections. That does not settle anything and it does not scale.

I found this out the direct way. A control framework for the AI Act reached roughly 194 controls in prose, and I put it through two independent audits. The first said it was fit to build on. The second found four Critical defects. All four turned out to be correct:

Article 113 — the general application rule "It shall apply from 2 August 2026" had been missed entirely, so the framework's whole "readiness tool" framing was wrong. Much of the Act is already in force.
Article 111(2) — legacy systems are out of scope unless subject to "significant changes in their designs", which is a different test from the Article 3(23) substantial-modification one the framework had assumed. No control existed.
Article 73(9) — a seventh statutory reuse provision, limiting serious-incident reporting for entities already subject to equivalent Union obligations. Five controls over-stated their duties as a result.
Article 42 — two presumptions and a deeming provision, one of which removes a requirement entirely for providers in scope of the Cyber Resilience Act. No control existed.

Two careful readers, same text, opposite conclusions. The interesting part is why every one of those defects happened. Not one was a misreading. Every one was a gap in what had been read — a provision nobody had opened, that no control pointed at, and that nothing tracked as unread. That class of error is invisible to review, because review only examines what is in front of it.

So: build the mechanism.

What regcheck is¶

A parser and a checker, about 1,200 lines of Python, no dependencies. It reads the plain-text consolidated act as published by EUR-Lex, turns it into an addressable corpus, derives an inventory of obligations, and verifies a control set against it.

python regcheck.py build  ai-act                        parse the act, derive obligations
python regcheck.py duties ai-act                        list every operator duty
python regcheck.py duties ai-act --uncovered ctl.json   list what has no control
python regcheck.py show   ai-act art_73/9               print any provision verbatim
python regcheck.py check  ai-act controls/ai-act.json   verify — exit 0 means green

The rule it enforces: nothing ships that the harness has not passed, and anyone can run the harness.

Provision IDs are the spine¶

Everything is addressed by ID, never by prose reference, so a citation cannot silently point at nothing.

32024R1689:art_9          32024R1689:art_9/2        32024R1689:art_9/2/a
32024R1689:anx_IV         32024R1689:anx_IV/2       32024R1689:anx_IV/2/g

This is the whole trick. "Article 9(2)(a)" in prose is a string that nobody validates. 32024R1689:art_9/2/a either resolves against the parsed act or it does not, and the difference is an exit code.

Integrity gates: verifying the parser, not just the controls¶

The checks below verify a control set against the corpus. But a corpus that has silently dropped legal text is worse than no corpus at all — every downstream check then passes against text that is not there, and passes confidently. So the parser is verified first, and build refuses to write a corpus that fails.

Gate	Question	What it caught
Reconstruction	Does every substantive source line survive into some provision?	A `SECTION` heading was overwriting the last paragraph of the preceding article. Fourteen provisions destroyed, including Article 27(5) and Article 15(5).
Enumerator count	Did every `(a)` and `3.` in the source produce a node?	Structural collapse that reconstruction cannot see.
Collision	Was any provision overwritten by another with the same id?	Annex VIII had lost items 1–6 of Section A; Article 43(1) had lost a two-point list.

The second gate exists because the first turned out to be weaker than it looked, and finding that out is the most useful thing in this section. Disable the paragraph rules entirely and every source line still reconstructs — the text is simply absorbed into the parent article. No loss, total structural collapse, invariant silent. Counting enumerators closes it: every (a) in the operative text is a node the parser owes you, so a deficit is a structural failure however intact the prose looks. On the AI Act both sides read 1262. Disable the paragraph rules and the node count drops by 654.

An invariant nobody has seen fail is not an invariant

selftest.py exercises every gate twice — against the healthy parse, which must pass, and against a deliberately sabotaged parser, which must fail. The sabotage cases are the real defects, kept as regressions.

This is not ceremony. Written naively, the reconstruction gate passed all four of its own negative tests, which is how the weakness above was discovered rather than shipped.

The seven checks¶

Check	Question it answers
`CITE`	Does the cited provision exist?
`REPEAL`	Is it still in force, or did an amending act repeal it?
`QUOTE`	Is the quoted text verbatim, or was it paraphrased?
`MODAL`	Does the control's obligation level match the provision's own verb?
`QUAL`	Does the control drop a qualifier that narrows the duty — including one inherited from a chapeau?
`COVER`	Does every operator duty have a control, or a written exemption?
`DUP`	Do two controls cite the same provision at different obligation levels?

Errors fail the run and set a non-zero exit code. Warnings are judgement calls that need a human, and are recorded either way.

REPEAL deserves a note. Repeals are derived by set difference between the base act and the consolidated act, not by reading the deletion markers in the text. The marker tells you that something went, not what. The difference matters because "cites a provision that was deleted" and "cites a provision that never existed" are different defects with different fixes.

The control schema¶

Controls are JSON, so they are diffable, reviewable and machine-checkable.

{
  "id": "DAT-4.1",
  "domain": "DAT",
  "obligation": "MUST",
  "roles": ["P"],
  "statement": "Where special categories of personal data are processed ...",
  "provisions": [
    {"id": "32024R1689:art_4a/1/a",
     "quote": "the bias detection and correction cannot be effectively fulfilled by
               processing other data, including synthetic or anonymised data"}
  ],
  "evidence": "Documented determination that alternatives were considered and rejected."
}

The quote must be verbatim — that is the point of it. It sits beside the control so a reader sees the source words next to the requirement and does not have to trust anyone, including the tool.

Two escape hatches, both deliberate and both visible. derived: true marks a control we added that the regulation does not require — it is then exempt from CITE, but flagged if it also cites provisions, because it cannot be both. And an exemptions.json maps a provision to a written reason it needs no control. An exemption is a reviewable decision; a silent absence is a defect. The tool's job is to make sure you never get the second when you meant the first.

Adding another regulation is a data change¶

Nothing in the parser, the obligation builder or the checker knows anything about the AI Act. A new regulation is an entry in sources.json:

"nis2": {
  "celex": "32022L2555",
  "consolidated": "../regulations-latest/nis2/....txt",
  "base": null,
  "scope": ["art_"],
  "roles": {
    "essential_entity": "\\bessential entit(y|ies)\\b",
    "important_entity": "\\bimportant entit(y|ies)\\b"
  }
}

The roles field is the dangerous one

It is the act's own vocabulary for the parties it binds. Get it wrong and the obligation inventory comes back near-empty — which looks exactly like a clean bill of health.

GDPR parsed with the AI Act's roles returns 1 operator duty. With controller and processor it returns 118. Always sanity-check the duty count against the size of the act before trusting a run.

So: does it work on any regulation? No — and the honest version of that answer is the point. It parses the EU AI Act and the GDPR with every integrity gate green: zero lines lost, every enumerator accounted for, 1262/1262 and 757/757. Run it against the Digital Omnibus, an amending act, and the enumerator gate fails at 191/202. Amending acts nest quoted replacement text inside numbered instructions, so Article 1's own (1)–(43) sequence collides with the (1) and (a) of the text it is quoting, and eleven instructions end up without a node.

That limitation is now declared in sources.json with a written reason, and the deficit still prints on every run. A source can be marked "parse_integrity": "advisory", which stops a known limitation masking a new defect — it does not hide it.

The useful claim is therefore not "it works on any regulation". It is: it tells you, mechanically and before you build anything on it, whether it works on yours. For audit purposes that is the more valuable property, because the alternative is not a tool that always works — it is a tool that fails quietly.

The discipline the tool enforces¶

The tool is half of it. The other half is a working method, derived directly from how the original defects happened.

Read the whole article, not the paragraph you think you need. Every confirmed error came from inferring across a gap, against text sitting in a local file that nobody had opened.
Never write "confirm before the build." Confirm now, or leave the control out. A deferred check is a defect with a note attached.
Derive counts mechanically. len(records), never prose. Every domain migrated so far had a wrong count in its own header — one said "12 controls" above sixteen headings, another said "three sub-domains, 15 controls" above two sub-domains and nineteen.
Never assert a negative from a script you have not tested on a known positive case. I nearly repeated the original sin here: checking whether Article 9 had been amended, my first script matched art_9 against art_95, art_96 and art_99. I validated the corrected version against a known-amended article before trusting its answer.
Inventory what you have not read. Every handover carries a read-register of articles opened in full. The four Criticals were missed because nothing tracked that they had never been opened.

Planning agent work against a legal text¶

The five rules above were written for a human. They matter more with an agent, because an agent shares the failure mode that produced every original defect — confident, fluent output over text it has not read — and adds one of its own, which is that it will cheerfully invent a citation that looks exactly like a real one.

The useful insight is that you cannot fix this by asking the agent to be careful. Instructions to be thorough are unfalsifiable; the agent believes it was thorough, and so does the next one. What works is making the claims checkable by something that is not the agent.

Give the agent a verifier it cannot argue with. check returns an exit code. An agent can write a persuasive paragraph explaining why a citation is substantially correct; it cannot make a non-existent provision resolve. This inverts the usual review burden — instead of a reader trying to spot what is wrong, the tool enumerates it, and the agent's job becomes clearing a list.

Make quotes mandatory. Requiring a verbatim quote beside every citation is the single highest-value constraint. A paraphrase is where hallucination hides; a quote either appears in the parsed text or it does not. It also means a reader who distrusts both the agent and the tool can check by eye.

Make "not read yet" a first-class artefact. Each session's handover carries a read-register — the articles opened in full — and an explicit list of what remains unopened. Without it, an agent's context window silently becomes the boundary of what it believes exists, which is precisely how four Critical provisions went missing. "I have not read Articles 42, 73, 111 or 113" is more valuable than any summary of the ones it did read.

Work in small, checked increments. One domain at a time, check after each, never batching. When something breaks you know which change did it, and an agent picking the work up later inherits a green baseline rather than an unknown one.

Have the agent report counts from the artefact, not from prose. Every domain header so far has carried a wrong count, and each was faithfully repeated forward until something computed len(records).

Assume the tool is wrong too. Four parser defects were found by treating surprising output as a bug in the harness rather than a fact about the law. When the duty count moved unexpectedly, the right move was always to find the specific row that changed and explain it — not to accept the new number because the tool produced it.

Written down, the method sounds like ordinary engineering discipline. It is. The point is that regulatory analysis has not usually been treated as something that admits of a test suite, and it turns out to be.

What it has actually caught¶

Worth being concrete, because a verification tool that has never found anything is not evidence of quality.

Five parser defects, each of which would have corrupted the migration silently. Four were found by accident, which is not a property audit tooling should rely on — the integrity gates above exist so the fifth was found by construction, in seconds, and so the class cannot recur.

The worst: a SECTION heading did not reset the parser's write target, so the section title overwrote the last paragraph of the preceding article. Fourteen provisions had their text destroyed. Article 27(5) was stored as the string "Notifying authorities and notified bodies". So was Article 15(5), the security-resilience provision, and Article 73(11) on serious incidents — meaning two later domains would have been built on text that was not there.

Also: (i) parsed as a Roman numeral rather than the ninth letter, so eight lettered lists broke at (i) and provisions like art_59/1/i did not exist. And Article 3's definitions parsed as one undifferentiated blob, making art_3/23, art_3/49/c and art_3/66 uncitable — all of which later controls needed.

The fifth was silent overwriting. Provisions were registered into a dictionary that replaced on key collision, so where one parent held two enumerated lists the second deleted the first without complaint. Annex VIII lost items 1–6 of Section A; Article 43(1) lost the two-point list offering the internal-control route under Annex VI. Sixteen collisions in total. The fix re-keys a restarted list onto an explicit subparagraph — art_43/1/sub2/a — so nothing is discarded and the id says where it came from.

That one had a consequence worth stating. With Annex VIII parsing properly, the repeal diff now reports anx_VIII/B/7 and anx_VIII/B/9 as deleted — and Article 49(4)(b) still cross-refers to "Section B, points 1 to 5, and points 8 and 9 of Annex VIII". A live citation to a deleted point, in force text. That is exactly the kind of defect the framework's transcribe-and-flag rule exists for, and until the annex parsed correctly it could not be checked at all.

A coverage denominator that excluded the entire high-risk requirement set. Chapter III Section 2 is drafted in the passive throughout — "Training, validation and testing data sets shall be subject to..." — so no party is named and the role detector found nothing. Twenty-four provisions across Articles 8 to 15 sat outside the denominator. A framework omitting the whole of Article 10 would have passed COVER. Article 16(a) supplies the missing subject, requiring providers to "ensure that their high-risk AI systems are compliant with the requirements set out in Section 2", so that attribution is now declared as data, naming the provision it relies on. The count moved from 190 duties to 220.

And one defect in the framework, found by the tool rather than a reader. A control cited Article 43(3) for a routing rule. QUAL flagged "only if" and "where applicable" as narrowing terms present in the provision but absent from the control. Following that up surfaced a third subparagraph nobody had addressed: a manufacturer may use the cheaper non-third-party conformity assessment route only if harmonised standards covering all Section 2 requirements have also been applied. That is commercially material, and no amount of re-reading the control would have found it, because the control looked fine on its own terms.

What a green run does not mean¶

This matters more than what it does mean.

A green run means the parser accounted for every line of the act, the citations are real, the quotes are verbatim, every operator duty has a control or a written exemption, and no qualifier was silently dropped. It does not mean the control is well drafted, that the evidence is the right evidence, or that the framework is fit for its purpose.

One distinction is worth being precise about, because it is where the reasoning nearly went wrong. The integrity gates prove that text survived into the corpus and that the expected number of structural nodes exists. Neither proves the text was attached to the right node. No purely textual invariant can: attribution correctness is what reading the whole article, and the verbatim quote sitting beside each control, are for.

The known limitations, stated plainly rather than left for someone to discover:

MODAL is advisory. It takes the strongest deontic verb in a provision without distinguishing a shall in a subordinate clause. Most remaining warnings on the AI Act set are exactly this — "risks that may emerge" is not a permission.
bound is cue-based. A provision whose subject is only implied can be missed. That is what the Chapter III finding above was, and it is not the only instance: 104 provisions act-wide carry a duty verb with no detected party, and they have not been individually audited. The Article 5 prohibitions are among them. This is now surfaced rather than buried — build prints the count, and regcheck.py unbound lists them by chapter. They are outside the coverage denominator, which is precisely the number an auditor should be shown rather than left to infer.
COVER counts a duty as covered if any control cites it or a descendant. It does not verify the control actually addresses the duty. QUOTE and human reading carry that.
Nothing reads recitals, and annexes are citable but excluded from the AI Act's obligation denominator, because in that act they specify document contents rather than impose duties. That is a per-source decision and it is recorded as one.

It removes the class of error that produced every finding in both audits. That class happens to be the one that cannot be caught by reading, which is why it is worth automating — but it is one class, not all of them.

Where this is going¶

The AI Act control set is being migrated from prose into the schema one domain at a time, with a check run after each. Four of eleven domains are done. Every migration so far has produced controls the prose did not have, and corrections to ones it did.

After that, the same treatment for the DORA and GDPR legs, and the cross-mapping between them — which will need its own checks, since a mapping has two endpoints and both have to resolve.

Sources¶

Regulation (EU) 2024/1689 (EU AI Act), consolidated text — EUR-Lex
Regulation (EU) 2016/679 (GDPR), consolidated text — EUR-Lex

All provision text quoted above is taken from the consolidated texts as parsed, not from secondary sources or from memory.