Metadata management: what to document for each data asset and how to keep the catalogue alive

Without metadata, a data asset is just another file. With it, it is a governed asset: you know what it means, where it comes from, who is accountable for it, who uses it and what restrictions apply, and quality, security and traceability policies can be applied to it. The data catalogue is the tool where that metadata lives; metadata management is the discipline that makes the catalogue reflect reality.

The four types of metadata

Following DAMA-DMBOK, with a fourth category for what is specific to each organisation's governance model:

Type	What it describes	Examples
Business	The functional meaning	Definition, glossary terms, rules, owner, purpose
Technical	The structure and build	Schema, data types, source system, load processes, dependencies
Operational	Behaviour in production	Refresh frequency, status, quality, incidents, usage
Governance	Model-specific attributes	Classification, personal data, criticality, authoritative source, use by AI systems

Mandatory metadata: the minimum to publish

The usual mistake is trying to document everything from day one. A short set of mandatory metadata, whose absence blocks go-live, works better:

• Asset name, readable by people.
• Functional description.
• Functional domain and the data product it belongs to.
• Data owner.
• Source system.
• Linked glossary terms.
• Refresh frequency.
• Information classification and presence of personal data.

That answers the essentials: what it is, who is accountable and how it is handled.

Extended attributes: what the tool does not provide out of the box

Tools such as Microsoft Purview or OpenMetadata cover technical metadata well, but not the metadata specific to each organisation. These are added as custom fields with controlled values:

Attribute	Values	Status
Information classification	Public / Internal use / Confidential / Restricted	Mandatory
Personal data	No personal data / Ordinary / Special categories / Pseudonymised	Mandatory where applicable
Data type	Master / Reference / Transactional / Analytical	Mandatory
Authoritative source	Yes (with approval date) / No	Where applicable
Source type	Business application / File / SharePoint / Web service / Sensors	Mandatory for integrated sources
Criticality	Critical / High / Medium / Low	Recommended
Use by AI systems	List of systems	Mandatory where applicable

Two important points. The tools' automatic classification (national ID numbers, IBAN, email…) is a complement, not a substitute: the official classification is the one approved by the data steward. And the “Use by AI systems” attribute is increasingly necessary: it shows which data feeds each assistant or model and helps prioritise its quality.

The metadata lifecycle

The golden rule: metadata is created when the asset is delivered, not afterwards. Each data product includes its metadata as a development requirement, so no documentation debt builds up.

1. Creation, during development, from the deliverables: requirements specification, functional design (glossary and users with access), technical design (tables and fields) and process inventory (flows and dependencies).
2. Validation by the data steward before go-live.
3. Publication in the catalogue by the custodian, as part of the publication checklist.
4. Maintenance with every relevant change: schema, owner, uses or sensitivity.
5. Annual review, especially of governance metadata (owner, classification, criticality), which goes out of date without anyone noticing.

Lineage: where each piece of data comes from

No asset is truly governed unless its origin, transformations and uses are known. Tools generate automatic lineage only for some sources; transformations in code (Python scripts or notebooks) usually require a manual procedure:

1. Inventory the complete flow, from the source systems to the reports, models, APIs or AI systems that use it.
2. Gather the technical evidence: code, schemas and a sheet per process showing which columns each output comes from.
3. Translate it into source–target pairs, per asset and, where feasible, per column.
4. Register in the catalogue whatever the scan does not detect and create the manual lineage relationships.
5. Describe the purpose of the flow and its rules in plain language.
6. Validate it functionally and approve the coverage before publishing.
7. Keep it up to date with every change.

Who does what

Role	What they do with metadata
Data owner	Knows the definitions in their domain, provides judgement and escalates discrepancies
Data steward	Validates definitions, descriptions and glossary, and provides the context of the source system
Data custodian	Loads and updates metadata in the catalogue, including lineage and extended attributes
Data architect	Defines cataloguing standards, extended attributes and controlled lists
Data governance lead	Safeguards the framework, monitors the minimums and produces the reports

How to write good descriptions

Tables. Describe the purpose and content without repeating the name. Bad: “Contains the streets”. Good: “Official street directory of the municipality, with the INE code of each street and its numbering range”. Better not to mention source tables, which change and leave the description out of date.

Fields. An explanatory description, whether it is part of the primary key, and a data type consistent with the content: a postcode is text, not a number, or it will lose its leading zero.

Indicators. Every description answers five questions:

Element	Example: average time to grant minor works permits
What is measured	Days from the complete application to notification of the decision. It indicates how responsive the service is
How it is measured	Sum of calendar days between both dates divided by the permits decided in the period
Unit	Calendar days
Thresholds	Target: 30 days or less. More than 45 days: alert
Interpretation	The lower, the better. A sustained increase points to a lack of resources or a bottleneck

How to tell whether it works

Four coverage indicators, which can be obtained from the catalogue tool itself through its API and reviewed by the data governance committee:

• Percentage of assets with complete mandatory metadata.
• Percentage with classification and personal data identified.
• Percentage with linked glossary terms.
• Percentage with a formally assigned owner.

An example of a complete record

This is what a dataset's record in the catalogue would look like:

Metadata	Value
Name	Official street directory
Description	Streets of the municipality with their INE code, street type, numbering ranges and district. It is the reference for validating addresses in all systems
Domain / product	Population and territory / Territorial reference data
Owner	Head of the statistics department
Steward	Cartography officer
Source system	Territorial management application
Glossary terms	Public road, District, Section
Frequency	Weekly
Classification	Public
Personal data	No personal data
Data type	Reference
Authoritative source	Yes (approved by the data governance committee)
Criticality	High
Use by AI systems	Customer service assistant (address validation)

With this record, anyone knows what it is, whether they can use it, whom to ask and which systems would be affected if it changes.

The business glossary: the other half of the catalogue

The catalogue describes assets; the glossary defines concepts. Without a glossary, two departments can call different things an “active customer” and their reports will never reconcile. To build it:

1. Start with the concepts that cause the most debate: customer, contract, case file, active employee, start date.
2. One definition per term, in plain language, validated by all the domains that use it.
3. Link each term to the datasets that implement it: that way you can see which tables correspond to each concept.
4. Subdomain-specific terms where necessary, inheriting the common ones from the domain.
5. One person accountable per term, usually the domain owner, who approves changes.

Information classification: the levels

Classification is mandatory and translates into real controls; it is not a paperwork exercise:

Level	Examples	Handling
Public	Open data, published indicators, regulations	No restrictions
Internal use	Operational documentation, internal indicators	Organisation staff only
Confidential	Data on specific customers or citizens, contracts under tender	Access by role and need to know, with access logging
Restricted	Health, minors, victims, security configurations	Minimal access, reinforced controls and auditing

Encryption is not a level but a control applied to confidential or restricted assets according to the risk analysis. And each dataset also states whether it contains personal data and of what type, because that determines the legal basis, retention and the need for an impact assessment.

Metadata and integrated sources

When a new source is added to the data platform, the catalogue must also record:

• the source type (business application, file, document management system, web service, sensors);
• the integration pattern (batch load, event stream, on-demand query, change data capture);
• the data access agreement with the owner of the source system: which data, how often, which schema, what sensitivity and what happens when the source changes;
• the complete lineage through to the products that use it.

That way, when a source system changes, you know within minutes which reports and models will be affected.

Metadata for artificial intelligence

AI systems, especially those that search an organisation's own documents (RAG), depend on metadata that used to be optional:

• Document validity: an assistant that cites a repealed regulation gives wrong answers with complete confidence.
• Classification and permissions: they determine what the assistant can see and on whose behalf it answers.
• Authoritative source: which document prevails when two say different things.
• Use by AI systems: to prioritise the quality of those assets and assess the impact of changes.

How to roll it out in phases

1. Phase 1: the minimum on what is critical. Mandatory metadata and classification on the most used datasets and those feeding AI systems.
2. Phase 2: glossary and lineage. Key business terms and lineage of the main data products, including manual lineage where automatic lineage does not reach.
3. Phase 3: extended attributes and automation. Criticality, authoritative sources, coverage indicators obtained via API and annual review.

Each phase closes with the coverage indicators, not with the tool installed.

Common mistakes

• Documenting afterwards: documentation done later never arrives, or arrives out of date.
• Copying the technical name as the description: “TB_CLI_001: customer table” helps no one.
• Relying only on automatic scanning: it detects tables, not meanings.
• A catalogue with no owner: if nobody reviews the governance metadata, within a year the owner of half the assets will no longer work there.
• Trying to catalogue everything at once: 100% of what is critical is better than 30% of everything.

At Bidatia we implement data catalogues with Microsoft Purview and OpenMetadata, and we design the metadata model that makes them useful. For the framework of roles and bodies, see data governance: roles and organisation.