Specification
ODF is a set of interconnected specs, grouped into domains that build on top of one another - from the structure of a single dataset, to how nodes process, exchange, and protect data.
Core Requirements
Functional
-
Complete historical account
The history of all data observed by the system is preserved - discarding or modifying data is akin to rewriting history.
-
Reproducibility
All transformations are fully reproducible, and any process can refer to data frozen in time.
-
Verifiability
Any party that plans to use some data can verify its validity.
-
Provenance
Any individual data cell can be traced back through all transformations to its ultimate source.
-
Privacy
Sensitive data can be exchanged and used in computations without disclosing it to unauthorized parties.
Non-functional
-
Transparency
The method is developed in the open, and is easy to understand and reason about.
-
Timeliness
Data propagates through transformation pipelines with minimal delay.
-
Transactionality
All operations have ACID properties and exactly-once semantics.
-
Decentralization
No entity has special privileges over data or can circumvent the above requirements.
Index
Datasets
How a dataset is structured, identified, hashed, and stored, and how external data enters it.
Logical Structure
General structure of an ODF dataset
Identity
did:odf identifiers, aliases, and references
Data Model
Event model, system columns, retractions and corrections, schema evolution
Data Schema
Logical schema format of data
Metadata Chain
Blocks, events, consistency rules, and block hashing
Data & Checkpoint Hashing
Physical and logical hashing of data and checkpoints
Storage Layout
Layout of dataset components in various storage systems
Ingestion
Sources, ingestion phases, and merge strategies
Compactions WIP
Merging data files and permanently removing history for compliance
Transformations
How derivative datasets are computed from their inputs in a reproducible, verifiable way.
Processing Model
Derivative datasets, watermarks, checkpoints, and query evolution
Engine Contract
Communication between coordinators and engines
Projections WIP
Temporal projections and collapsing changelog streams into state
Exchange
How datasets and query results move between parties that don't need to trust one another.
Transfer Protocols
Simple and Smart Transfer Protocols for synchronizing datasets
Verifiable Batch Queries
Commitments that make query results verifiable
Query Protocols WIP
Standard query interfaces and federated query planning
Discovery WIP
Dataset registries and anchoring dataset state to external ledgers
Automation
Declarative configuration of nodes and scheduling of the work they perform.
Resource Framework
Declarative resource model and manifest format
Tasks and Flows
Scheduling and execution of workloads
Event Bus WIP
Delivery mechanism and common types of events in the system
Webhooks WIP
Delivery of dataset events to external HTTP endpoints
Auth & Privacy
Who can access resources and data, and how sensitive data stays protected.
Authorization
Accounts, groups, auth attributes, and policies
Decentralized Permissions WIP
Delegated access control spanning nodes and organizations
Layered Encryption WIP
Encryption of data and metadata, key management and rotation
Confidential Computing WIP
TEEs, ZK proofs, FHE, and privacy-preserving transformations
Design Decisions
-
Data is a history of events
Data is modeled as immutable, ever-growing streams of events instead of snapshots that lose information with every update.
Data is a set of events, observations, or propositions believed to be true at a certain time.
-
Trust through verifiability
Transformations are deterministic queries executed by sandboxed, strictly versioned engines - so anyone can repeat them and verify the results, synchronously or long after the fact.
-
Stream processing
Queries are defined once and run forever, producing the same results whether data arrives monthly in bulk or as a real-time stream.
-
Metadata as a passport
An append-only, cryptographically linked metadata chain records where data came from and everything that ever influenced it.
-
Strong transactional semantics
All coordination operations are ACID, and all transformations have exactly-once semantics.
-
Storage-agnostic exchange
Only metadata needs to be trusted - data can be downloaded from any untrusted source and verified against hashes in metadata.
Evolving the Spec
The specification is developed in the open. Changes are proposed and discussed as RFCs, and every spec links to the RFCs that shaped it.
Latest RFCs
Explore the latest Request for Comments shaping the future of Open Data Fabric protocol.
Authorization Model
Defines the authorization model for ODF nodes and how accounts, groups, auth attributes, and policies are expressed within the Resource Framework.
IaC Task and Flow System
Proposes a new set of resources to define and control workload scheduling and execution within ODF nodes.
Infrastructure-as-Code Resource Framework
Proposes a new Open Data Fabric manifest format and a set of resource types aiming to re-center ODF around the declarative approach for defining the desired state of the system.