Data governance
8 min

Metadata Management Framework: Four Stages and a Self-Assessment

Key takeaways

  • A metadata management framework is the set of decisions that makes metadata management repeatable: scope, roles, standards, the record, the approval workflow, the serving interface, and the measures.
  • The four stages are Collected, Described, Attested, and Served. Each is defined by what an AI system can do with the record, from finding an asset to acting on an approved definition with its provenance attached.
  • Attested, or SME-approved, is the stage published frameworks skip. Automation gets an organization to stage two. Stage three needs a named person to approve each consequential meaning, with a date, a version, and a rule.
  • The twelve-statement self-assessment is run per domain. Your stage is the highest one where all three of its statements are true, and your next job is the first false statement after it.
  • Programs stall at stage two for three reasons: nobody owns the meaning, the meaning is held as prose, and approval is scoped to assets instead of concepts. All three are decisions, and no scanner makes them.
In this article
Share

A metadata management framework is the set of decisions that makes metadata management repeatable. It says which metadata is in scope, who owns each definition, and what form a governed record takes. It says how a proposed meaning becomes an approved one, how the approved record reaches the systems that use it, and how you know the whole thing is working. The framework in this post has four stages. Each stage is defined by what an AI system can do with your metadata once you are there. A self-assessment table places one domain of your organization in about ten minutes.

Published frameworks for metadata tend to share one shape: a ladder of automation. The bottom rung is an inventory. The top rung is metadata that updates itself, and the word for the top is usually "active" or "autonomous". That ladder answers how fast your metadata moves. It never asks whether the meaning at the top was approved by anyone. The four stages below put that question in the middle, because an agent needs it answered before it acts.

The four stages are Collected, Described, Attested, and Served. Your stage is the highest one at which every statement in the self-assessment is still true. Your next job is the first false statement after it.

What is a metadata management framework?

A framework is different from a strategy and from a tool. The strategy says why the organization is doing this. The tool is where the work happens. The framework sits between them and holds the decisions both depend on.

The seven components, and the test that each one exists

ComponentThe decision it holdsThe test that it exists
ScopeWhich domains, systems, and terms are in the program, and in what orderA written list of the terms whose wrong meaning costs something
RolesWho proposes, who approves, who consumes, and who inherits when the approver leavesEvery governed term resolves to a named steward with a successor on file
StandardsThe form a record takes, so every tool reads the same thingRecords are held in RDF, SKOS, OWL, and SHACL, and export in full
The recordThe fields a governed definition must carry before it countsDefinition, scope, steward, status, dates, version, mappings, validation rule, and provenance
WorkflowHow a proposed meaning becomes an approved one, and when it is reviewed againA queue where automation proposes and a person confirms, with a review interval on every approval
ServingHow consumers, including agents, get the approved recordOne interface that returns the definition with its approver, date, and version attached
MeasuresHow you know the program is workingThe share of consequential terms with a current approval, and the age of the oldest expired one

In our post What Is Metadata Management? we state that the discipline has four jobs: collect, define, govern, and serve. The components above make those four jobs repeatable across domains. The stages below are what the organization looks like as each job gets done properly.

What are the four stages of metadata management maturity?

The stages are cumulative. Each adds one thing a machine could not do before. The names describe the state of the record, since the record is what an agent reads.

The four stages, defined by what the record holds

StageWhat the organization can sayWhat the record holdsWhat an AI system can do with it
1. CollectedWe know what data exists and where it livesTechnical metadata harvested by scanners: schemas, tables, columns, types, refresh timesFind an asset. Infer everything else
2. DescribedWe know what most of it is called and roughly what it meansDescriptions, glossary terms, tags, and inferred lineage, usually as prose, with nothing that separates a guess from a decisionRead a plausible meaning, with no way to tell whether it is approved or expired
3. AttestedFor the terms that matter, we know who approved the meaning and whenA structured definition with scope, a named steward, status, dates, a version, column mappings, and a validation ruleResolve a term to ground truth, cite the approver, check a value against the rule, and refuse a record whose review date has passed
4. ServedEvery consumer reads the same approved record at the moment it asksEverything in stage three, exposed through one interface with provenance attached, and the catalog syncing from itAct on the record in production, show its basis to a person, and leave an audit trail a regulator can query

Attested, or SME-approved, metadata is the stage that published ladders leave out. Our post Ground Truth: SME-Approved vs Inferred Metadata covers the difference between a description a crawler guessed and one a named expert confirmed. Stage two can be reached by buying a catalog and running its scanners. Stage three cannot. It requires a decision about who owns each meaning, and a record that carries the decision.

Stage four is where the framework proves it was real. Our post Active vs Passive Metadata: Why a Well-Organized Catalog Still Goes Unused argues that movement alone is the wrong measure of maturity. Stage four is movement of the right thing: the approved record, with its approver and its date attached.

How do you assess which stage you are at?

The self-assessment below has twelve statements, three per stage. Mark each one true or false for one domain, since stages vary by domain and an average hides the gap. Your stage is the highest one at which all three of its statements are true.

The metadata management self-assessment

#StatementStage
1We can list every data asset in scope and the system that holds it1. Collected
2Technical metadata for those assets is harvested automatically and refreshed on a schedule1. Collected
3A new asset appears in the inventory without anyone filing a ticket1. Collected
4Most assets in scope carry a description a person can read2. Described
5A business glossary exists and its terms are linked to at least some assets2. Described
6Lineage can be drawn for the assets that feed regulatory or financial reports2. Described
7Every term whose wrong meaning carries a cost has one approved definition, a named steward, and a review date3. Attested
8The approved definition is structured data, with mappings to physical columns and a rule that tests values against it3. Attested
9A field on the record separates approved definitions from descriptions, so a consumer can decline the unapproved one3. Attested
10An agent or application can request the approved record and receive it with its approver, date, and version attached4. Served
11Our catalogs display definitions synced from the governed record and hold no competing version4. Served
12"Who signed this off and when" is answered by a query, for any term, including past versions4. Served

If seven is your first false statement, you are on the stage-two plateau. Our post Metadata Quality: How to Measure It and What to Do About It turns statements seven to twelve into metrics.

Why do programs stall at stage two?

Stage two is where a catalog's own scores look best and an agent's answers look worst. Coverage is high. Descriptions exist. Lineage renders. Nothing in any of it is approved. The three things that stop a program there are all decisions, and automation cannot make them.

Three reasons a program stops at stage two

  • Nobody owns the meaning. The catalog has an owner field, and it holds whoever created the table. Ownership of a definition, with the authority to approve it and the duty to review it, was never assigned. Our post Metadata Management vs Data Governance: What Each One Decides covers where that assignment comes from.
  • Meaning is held as prose. A definition typed into a description field can be read by a person and ingested by a model. It cannot be checked, versioned per concept, or mapped to forty columns at once.
  • Approval looks expensive. Fifty thousand columns and ten stewards will never meet. The mistake is scoping approval to the asset. Scoped to the concepts whose wrong meaning costs something, the work is a few hundred decisions. Suppose a crawler's description is right 85% of the time. 85% accurate isn't good enough for the terms an agent uses to approve a credit limit or file a report, and those are the terms stage three exists for.

How do you move from one stage to the next?

Each transition is one piece of work with a finish line. The order matters, because each stage produces the input the next one needs.

The three transitions

  1. Collected to Described. Connect the scanners and let automation draft descriptions, glossary links, and lineage. Record the source of every draft, so a model-written description is marked as one. Finish line: statements four to six are true.
  2. Described to Attested. List the terms whose wrong meaning carries a cost. For each one, name a steward, write the definition and its scope as structured data, map it to the physical columns, and attach a validation rule. AI proposes the drafts and the mappings, and your experts confirm them. Finish line: statements seven to nine are true for every term on the list.
  3. Attested to Served. Expose the approved record through one interface that returns the definition with its approver, date, and version. Point the catalog at it, so catalog descriptions sync from the governed record. Give consumers a rule for expired records: refuse, and route to the steward. Finish line: statements ten to twelve are true.

Our post Metadata Management Best Practices for AI-Ready Data sets out the practices that keep each stage from decaying, and we leave them there. The framework decides what the stages are and how you know which one you are at. The practices are how you stay.

How does the framework fit with governance, ontologies, and tools?

A metadata framework is one of several a data organization runs. Each one has a job.

What each neighbouring discipline decides

Where does TopQuadrant fit?

TopQuadrant is built for the move from stage two to stage four. Your catalog stays and keeps its discovery role. TQ Data Foundation sits above it as the governed source of meaning the catalog syncs from. Our metadata management use case page shows how TQ Data Foundation does this in practice.

In TQ Data Foundation, AI proposes definitions and mappings, and your experts confirm them. Nothing becomes authoritative without a person. Each approved record carries a named steward, a date it was last confirmed true, and a version. SHACL validation turns the definition into a test the data runs on load. Everything is held in RDF, SKOS, OWL, and SHACL and exports in full at any time. TopQuadrant helped author SHACL and RDF. We maintain 300 reusable standards-based models, so stage three starts from governed structure. The same governed record an analyst reads is what an agent resolves against, through the same interface. That is stage four.

Run the self-assessment on one domain this week. Find the first false statement. That is where TopQuadrant starts.

Give your AI the context it's been missing

See how the TQ Data Foundation turns your enterprise knowledge into trusted, Al-ready context.