2026-10-28 –, Naturalis 01
Do you need a simple way to manage FAIR data while still keeping rich metadata, provenance, and persistent identifiers?
In our light talk you will discover Metafiles, a lightweight approach designed for research and digitalization projects where many datasets and files require structured metadata and clear relationships. The system combines ARK persistent identifiers, structured XML metadata files, and a consistent RDF representation of metadata and stored efficiently in a relational database.
Instead of relying on large infrastructures, this approach focuses on practical workflows researchers can integrate directly into their projects. Metadata can be shared across related activities, dependencies between datasets, scripts, and results can be captured through machine-readable provenance, and responsibilities for data management can be organized hierarchically across teams.
After this light talk you will better understand how persistent identifiers, provenance, and shared metadata support scalable and reproducible research workflows while remaining simple enough for everyday research use.
Persistent identifiers
The system separates internal file references from external identifiers.
Inside the system, files can be referenced using simple relative names, while
externally all objects are identified using persistent ARK identifiers.
Different strategies for generating ARK identifiers are supported. These include
identifiers derived from content hashes (using a betacode encoding), identifiers
based on file names, and transparent human-readable identifiers when appropriate.
This flexibility allows institutions or projects to adopt identifier policies
that match their workflows.
The system supports delegated namespaces through ARK shoulders, allowing
sub-authorities to manage their own identifier spaces within a shared namespace.
An integrated ARK resolver is included as part of the infrastructure.
PID relationships and provenance
The system allows relationships between research objects to be expressed as a
PID graph. Relationships between datasets, files, scripts, and derived results
can be described using machine-readable metadata based on the semantics of
dcterms and PROV.
These relationships can be defined both for objects managed within the ARK
infrastructure and for external resources identified by other persistent
identifiers or URLs. This makes it possible to describe provenance chains
across heterogeneous research infrastructures.
For internally managed objects the system also supports automatic creation of
bidirectional relationships, bulk linking between sets of files, and structured
references between datasets and derived outputs.
FAIR data management
Metafiles supports both simple and complex metadata descriptions. Metadata can
include widely used vocabularies and ontologies such as FOAF or ORG when needed,
while still allowing lightweight descriptions for smaller datasets.
Projects can define policies for persistence of both data and metadata, and the
system can enforce or validate these policies. This helps ensure that metadata
remain accessible even when data are archived, moved, or removed.
The system can also support metadata requirements connected with regulatory
frameworks such as NIS2 where relevant.
Data organization
The basic management unit is a directory or an entire subtree of a file
repository. This reflects how research data are typically organized in
practice and allows metadata management to follow existing project structures.
Metadata can be attached to individual files, but more commonly they are defined
for directories, directory subtrees, or groups of files selected using filename
pattern matching.
Two types of metadata inheritance are supported:
- vertical inheritance, where metadata defined at higher levels of the
directory structure apply to files in lower levels, and - horizontal inheritance, where files may match multiple filename patterns
and therefore inherit metadata from multiple definitions.
Metadata extracted by external tools can be imported using simple XML files with
a clear structure.
Implementation
The system is implemented in Python and builds on several widely used open-source
libraries:
- rdflib for processing, serializing, and storing RDF metadata
- SQLAlchemy for storing metadata in relational database systems
- lxml for processing XML metadata and configuration files
Support for XML XInclude allows metadata and configuration fragments to be
distributed and reused easily across projects.
