*And I didn't scrape all of NIST to do so, here's why!* #### NIST Security Controls That Don’t Exist A while back I was researching NIST training datasets on HuggingFace.co. The largest ones tend to be massive, with hundreds of thousands of rows extracted from the NIST that are pre-chunked and ready to feed a model. Scrolling through one, I found rows labeled as security controls with IDs like HA-25, WE-12, and AM-6. These security controls and families do not exist. NIST SP 800-53 has twenty control families, AC through SR. HA, WE, or AM don’t fall anywhere within these control families. What I was looking at were extraction artifacts, meaning that the pdf chunker had misread fragments of document text as control identifiers and stamped them into the metadata. This is a predictable output of running automated extraction accross tens of thousands of pages, where no human could ever review each and every single row. The process simply has no way to alert you that HA isn’t a control family. I point this out not to be nitpicky about someone’s work but to highlight an issue that I see coming. People out there are building compliance on corpora like this right now without taking into account the implications of what they are doing. An assistant trained on fabricated control IDs is dangerous because it will actively hallucinate with citations. It will tell an ISSO confidently and with certainty that control HA-25 requires something. This is actually worse than an assistant that tells you “I don’t know” since we at least know that the assistant is being truthful with its limits In writing this, I’m not trying to call out specific datasets, projects or builders. The issue here is that there is a flaw baked into an approach that I want to address. To be absolutely upfront, my own pipeline produced the same class of defect during this buildout, and it was only a domain review that caught it. I’ll dig more into that later on. #### Information from 1989 - 2020 being held at equal weight The fake control IDs are the main and most visible issue, however there are structural ones that run deeper. The practice of scrape-everything corpora doesn’t appear to make any distinction between current and superseded guidance. Info from pdf files decades old can sit a long side NIST SP 800-53 with identical standing with nothing in the schema telling the model in training which one is relfecting guidance thats actually in force. A model trained on such corpus doesn’t learn NIST’s guidance per say, it learns NIST’s history. To be fair to the people building out these corpora, being comprehensive is thought about a lot when creating these datasets, the work of extracting this information is really hard and the popularity of these datasets prove that there is real demand for NIST info in ML pipelines before most of the compliance world was paying attention. I’d rather these exist than not but if we’re going to do this, we should do this right. This is the point where my day job and this side project intertwine. I’ve spent the majority of my career inside the Risk Management Framework (RMF) authoring System Security Plans, Security Assessment Reports, Plan of Action & Milestones, and many other authorization lifecycle artifacts, and a consistent thesis I keep returning to is something I call the **Authorization Gap.** The Authorization Gap is basically the idea that AI has dramatically industrialized capacity while our assurance processes haven’t quite moved. I think training data specifically is a perfect place to highlight this gap. Anyone would be able to pull a dataset and fine-tune a model, but who exactly signed off on what is in that data? Who would even be able to tell you? Training data has a supply chain and very few are even inspecting it. I essentially built the data set I wanted to see exist and shipped last weekend: **[rmf-ato-core](https://huggingface.co/datasets/ezesecops/rmf-ato-core)**, with the full pipeline public on **[GitHub](https://github.com/ezesecops/rmf-ato-core).** #### Curation So one of the first things you may notice is that corpus I shipped is roughly 5,510 rows. When compared to the massive corpora that can 100x larger, it’s significantly smaller but that is done intentionally. Here I’m betting on four properties that I think will make an assistant trained on it more effective: - Being small - Being current - Being structurally correct - Being traceable I want a much smaller corpus that an ISSM or ISSO can trust as opposed to one 100x larger that no human could have realistically have reviewed rigorously. The corpus has two tiers: Tier 1: The RMF spine made up of SP 800-37r2, 800-53r5, 800-53A, the 800-53B baselines, FIPS 199 and 200, 800-60, 800-18, 800-30, 800-39, 800-137. Tier 2: The NIST AI RMF, its Playbook, and SP 800-218/218A — the SSDF and its generative-AI community profile. I created and published this corpus because I couldn’t find anyone who published a corpus pairing the classic authorization documents with AI governance documents. I believe the pairing is a logical first step to a problem I think about a lot: **How do you authorize an AI system?** #### Parse Structure The decision to use machine-readable OSCAL JSON from [usnistgov/oscal-content](https://github.com/usnistgov/oscal-content) repository published by NIST does most of the heavy lifting in this project. This is because it allows every control, enhancement, and parameter to be received as structured data, thus making the output more trustworthy. If this data were chunked and pulled from PDFs via regex, this decision compounded over thousands of PDFs with zero human review is just a recipe for inaccurate outputs with citations. The PDFs still had to be parsed for the narrative documents: 800-37’s tasks, FIPS 199’s categorization tables, the AI RMF’s subcategories. But the pipeline enforces the rule that rows derived from PDFs can never carry a control ID, even when control IDs appear in the prose. And as belt-and-suspenders, the validation stage rejects any row whose control ID doesn’t match the whitelist of the twenty real Rev 5 families. The pipeline is not merely unlikely to publish control HA-25. It *cannot*. #### Treating a Dataset like a supply chain artifact This is the part of the project that’s most shaped by my day job. Every document in the corpus traces to a manifest — a single JSON file listing each source, its revision, its publication date, its URL, and why it’s in scope. The fetch stage records a SHA-256 hash of every retrieved artifact, and every published row carries the hash of the exact file it came from. Provenance ships with the dataset. And the piece I care about most: **the rejection log is a published, first-class output.** The pipeline rejected 1,131 chunks — withdrawn controls, front matter, bibliography entries, running-header fragments, an entire appendix of 2008 OMB memoranda with zero retrieval value — and every single rejection is in `rejections.jsonl`, in the repo and on the Hub, with a rule and a reason. The norm in dataset building is to silently drop whatever doesn’t parse. This one shows its work, the way an assessment report shows its work. That discipline forced one honest trade I want to state plainly rather than bury: 97 chunks from PDF page-break artifacts came out as mid-sentence fragments — real prose, genuinely damaged. I rejected them rather than publish them dirty, and the dataset card says so. Precision over recall, disclosed. A limitations statement for a dataset, exactly like the ones we write for systems. #### How it was built: a spec, an agent and five checkpoints I’ll be direct about the methodology, because I think it’s as instructive as the artifact: I did not hand-write this pipeline. I directed Claude Code through it, and the real engineering artifact of this project is the build specification I wrote before any code existed — the curation rules, the schema, the validation rules, the acceptance criteria, and five mandatory human-review checkpoints where the agent had to stop and show me its output before proceeding. The upload script refuses to run without an explicit approval flag, creates the Hugging Face repo private, and is incapable of setting it public. I flip that switch, not the tooling. My role was the quality gate, and it turned out to be the load-bearing one. The agent’s OSCAL parsing was excellent. Its PDF extraction needed an SME reading the output: I caught a glossary being mistyped as generic sections, made the calls on when a rejection rule was correctly filtering junk versus amputating recoverable content, and approved a document substitution when a newer revision turned out to have gone final. None of those were code problems. All of them were domain problems. And then shipping taught its own lessons, fast. My CI went red within minutes of the first public push: the agent had developed everything on Python 3.14, and a backslash inside an f-string expression — legal since 3.12 — is a syntax error on 3.11, which my repo claimed to support. “Works on my machine” applies to language versions too. Fixing it surfaced two details worth recording. First, `ast.parse(feature_version=(3,11))` will *not* catch that class of error — f-string tokenization happens before the feature check applies — so verifying compatibility required scanning f-string expressions directly. Second, sitting right at the failure site was a latent bug: a regex written as `r"\\s+"`, matching a literal backslash instead of whitespace, which had silently never worked. We verified the fix produced a byte-identical dataset before touching the frozen build. If you take one thing from this section: agents will happily build you 128 passing tests around a defect no test was written to see. The spec, the checkpoints, and a human who knows the domain are not overhead on the process. They are the process. #### Beyond one dataset EO 14028 made SBOMs a fact of life for software sold to the federal government. The same logic is coming for AI — the “AI bill of materials” conversation is already live in policy circles, and SP 800-218A gestures directly at provenance expectations for model development. Sooner or later, every authorizing official looking at an AI-enabled system will ask the question that today has no good tooling answer: *what is in the training data, and who vouches for it?* rmf-ato-core is one small worked example of what an answer could look like: a manifest, hashes, a validation regime, and a rejection log — assurance artifacts for data, built by someone whose job is assurance. I’m continuing to explore what that discipline looks like at the scale of a model hub. #### The Artifacts The dataset is live: [huggingface.co/datasets/ezesecops/rmf-ato-core](https://huggingface.co/datasets/ezesecops/rmf-ato-core). The full pipeline — manifest, build spec, tests, CI, and rejection log included — is at [github.com/ezesecops/rmf-ato-core](https://github.com/ezesecops/rmf-ato-core). If you work in this space, feel free to file issues. Corrections from RMF practitioners are especially welcome. SME review is the whole thesis, and that includes review of my work. If you actually made it this far, thanks for reading. I really appreciate it!