Maryk RocksDB Store: Architecture

This document explains how the RocksDB-backed data store is structured and how it maps Maryk’s versioned data model onto column families, keys, and values. It is written for contributors who want to understand the implementation.

• Module: store/rocksdb
• Key entry points:
  • RocksDBDataStore – lifecycle, request loop, migrations, column family setup
  • processors/* – request handlers (Add/Change/Delete/Get/Scan/Updates)
  • processors/helpers/* – low-level encoding/decoding and KV helpers
  • TableType, *TableColumnFamilies – column family descriptors and handles
  • Transaction, DBAccessor, DBIterator – local write buffering and reads

High-Level Overview

Maryk stores each DataModel in a set of RocksDB column families. Current (“latest”) values live in table, while keys, uniques, and indexes have their own families. When keepAllVersions = true, history is written into separate historic families with a custom comparator to keep “latest first” ordering per key.

All higher‑level request semantics (validation, update streaming, filtering, aggregates) are shared across engines and implemented in store/shared. The RocksDB module focuses on low‑level persistence and iteration.

Column Families per Model

For every registered DataModel (identified by UInt in dataModelsById), we create these column families:

• Model – stored schema and migration state for the model.
• Keys – existence + creation version per object key.
• Table – latest property values per object, plus a “latest version” marker.
• Index – latest secondary index entries.
• Unique – latest unique constraint entries.

If keepAllVersions = true:

• HistoricTable – all historical values per object.
• HistoricIndex – historical index entries.
• HistoricUnique – historical unique entries.

Column family names are compact byte arrays: first byte is the TableType (1..8), followed by the varint-encoded model id. See TableType.getDescriptor.

RocksDB options:

• All column families share an explicitly owned LRU cache. Default-owned DB options also attach a shared write-buffer manager and statistics; caller-supplied DB options retain ownership of those settings. RocksDBTuning provides resource budgets without changing the existing open signature.
• Maintenance iterators and record reads disable cache filling. Request snapshots retain normal cache behavior. Eligible bytewise prefix predicates use scoped native iterator bounds, while historic comparator predicates keep their existing boundary checks.
• Checked iterators report native status on exhaustion and before closing. Diagnostics expose approximate memory accounting, cache counters, compaction backlog and write stalls; checksum verification runs on the IO dispatcher.
• Column families use LZ4 compression, with Zstandard for the bottommost level. Each existing SST retains its own codec metadata; settings apply to new flush/compaction output without rewriting files at open. The bindings retain the codecs needed to read older Snappy and uncompressed files.
• Table and HistoricTable use a fixed-length prefix extractor sized to the model key for efficient key‑range scans.
• Historic* families use VersionedComparator, which sorts by key/qualifier first and version second, so qualifiers stay grouped.
• Presence-only index and migration-marker checks use exact keyExists reads, avoiding value copies. Transactional checks retain transaction reads for pending-write visibility, conflict tracking, and atomic work accounting. keyMayExist is not treated as proof of presence.
• Sequential readers explicitly enable RocksDB’s total-order seeks, including snapshot readers, so scans can cross model-key prefixes. Record-local readers retain prefix-restricted seeks.
• Transactional range/prefix deletion uses total-order iterators and checks iterator status before staging each bounded batch of deletes. This includes pending writes and supports savepoint rollback without relying on non-transactional range tombstones.

Keys, Qualifiers, and Values

Maryk uses a row/column style encoding. The primary key (Maryk Key) is followed by a qualifier that identifies the property (and collection item if applicable).

Latest data, in Table:

• Object creation: table[key] = version
• Latest version marker: table[key || [LAST_VERSION_INDICATOR=0b1000]] = version
• Soft delete flag: table[key || [SOFT_DELETE_INDICATOR=0x00]] = (version || [0x01])
• Property value: table[key || qualifier] = (version || value)

Creation version, in Keys:

• keys[key] = version

Historic data (when enabled):

• historicTable[key || qualifier || inverted(version)] = value
  • The version is stored in the key suffix with all bytes inverted to make newer versions sort first (forward scans yield latest‑first). See toReversedVersionBytes.

Versions are 8‑byte HLC timestamps. Helper functions live in processors/helpers/VersionBytesConverters.kt.

Indexes and Uniques

• Index (latest):

  • Key: index[indexRef || indexValueBytes || keyBytes]
  • Value: version

• Index (historic):

  • Key: historicIndex[indexRef || indexValueBytes || keyBytes || inverted(version)]
  • Value: empty

• Unique (latest):

  • Key: unique[uniqueRef || valueBytes]
  • Value: (version || keyBytes)
  • Unique names are enumerated by writing a single marker with a 0x00 prefix; see createUniqueIndexIfNotExists and getUniqueIndices.

• Unique (historic):

  • Key: historicUnique[uniqueRef || valueBytes || inverted(version)]
  • Value: keyBytes

Helpers for writing/erasing these entries are in processors/helpers/*IndexValue.kt.

Request Flow

RocksDBDataStore.startFlows() runs a coroutine actor that receives requests and dispatches to processors:

• Add → processAddRequest → validates values, writes keys/table, uniques, indexes, and emits an Addition update.
• Change → processChangeRequest → reads current values as needed, updates values and affected indexes/uniques, emits Change.
• Delete → processDeleteRequest → either hard-deletes all KV for the key (plus history) or writes a soft‑delete flag; emits Removal.
• Get/Scan → processGetRequest / processScanRequest → read values from table or historicTable depending on toVersion; use Keys to iterate keys efficiently; optional aggregation via Aggregator.
• GetChanges/ScanChanges/Updates → stream versioned changes based on fromVersion..toVersion using historic storage when available.

Common read helpers:

• DBAccessorStoreValuesGetter decodes values on demand and can capture the highest seen version while reading.
• getValue() selects latest versus historic storage based on toVersion.

Current projections of up to 16 simple root properties skip unselected collection subtrees with seeks on the existing request iterator. This avoids walking every collection item to return a few metadata fields. Missing fields, soft deletion, selected-field versions, snapshots and pending transaction writes retain their ordinary semantics. Small scalar models, full records, complex selections and historic snapshots use the existing sequential decoder. Historical projections also inspect unselected qualifiers to preserve their visible lastVersion.

Historical point reads reuse a supplied request reader for both present and absent values. A missing/deleted value or a decoder returning null does not create a second iterator or invoke the decoder again. Reader ownership stays with its caller.

Older stores can have a current soft-delete marker without its historic row. The change-reader fallback uses that marker only when its timestamp lies within the inclusive requested version window. Finite update history narrows this window to the single indexed revision, so a later deletion or restoration cannot consume an older revision’s page limit or become its continuation boundary.

For snapshot state and filters, a current marker whose timestamp is at or before the requested snapshot overrides an older historic marker. This applies to both deletion and restoration; a later marker cannot change an earlier snapshot. Change readers merge a recovered current transition with existing history and retain at most maxVersions soft-delete transitions, independently of other properties. Active-record history still includes a restoration transition when its historic row is missing.

Transactions and Concurrency

The store opens RocksDB as an OptimisticTransactionDB. Maryk keeps a small Transaction adapter around the native RocksDB transaction API so processors can use the same request-local access pattern while commits are handled by RocksDB atomically.

• Transaction.put/delete write into the native transaction batch.
• getForUpdate() delegates to RocksDB optimistic conflict tracking.
• commit() commits the native transaction atomically and starts a fresh native transaction for callers that process multiple objects in one request.
• Point reads and iterators during a transaction are provided by RocksDB and include pending writes.
• Creation-version probes use buffer-based point reads, preserving request order, duplicates, snapshot visibility, legacy historic-marker fallbacks, and transaction conflict/work accounting. Warm small-value probes did not justify batching on Native; stored metadata pairs still use MultiGet for coherent reads.

This keeps the implementation portable across JVM and Native targets while relying on RocksDB for atomic commit, rollback/savepoints, read-your-writes, and optimistic conflict detection.

Maryk’s public transaction scope is separate from this native adapter. Each preview replays its journal in a fresh native transaction and closes without publication. A datastore revision key protects reads and predicates across previews; commit publishes the complete journal and retry receipt together. Application suspension holds no native transaction open. AtomicRequest uses the same engine with initial-state checks. See the shared transaction contract.

Migrations and Version Updates

Operational details and full runtime API are documented in Migrations.

Stored model definitions, dependency payloads, and storage-format flags are read through one scoped RocksDB snapshot; model/dependency payloads are fetched together with MultiGet. Migration progress and target identity are also fetched together through scoped snapshot read options, which are closed before the snapshot is released.

At open time, RocksDBDataStore compares the stored model in Model with the provided definition:

• If models are equal → no work.
• If only safe additions or new indexes exist → it backfills index data (fillIndex) and stores the new model.
• If incompatible changes are detected → it calls migrationConfiguration.migrationHandler; the handler can perform corrective writes and must return:
  • MigrationOutcome.Success
  • MigrationOutcome.Partial
  • MigrationOutcome.Retry
  • MigrationOutcome.Fatal

Pass these settings under migrationConfiguration = MigrationConfiguration(...) from maryk.core.models.migration.

After a successful migration or a brand new model, versionUpdateHandler can run custom logic (e.g., seed data) before the new model is stored.

See: model/checkModelIfMigrationIsNeeded.kt and model/storeModelDefinition.kt.

Runtime operations:

• migrationConfiguration.migrationStartupBudgetMs + migrationConfiguration.continueMigrationsInBackground support startup handoff to background migration.
• Optional phase hooks also live under migrationConfiguration: migrationExpandHandler, migrationVerifyHandler, migrationContractHandler.
• Pending models are request-blocked via assertModelReady.
• Operators can inspect/control with pendingMigrations, migrationStatus(es), awaitMigration, pauseMigration, resumeMigration, cancelMigration.
• Default lease is local (RocksDBLocalMigrationLease) to avoid duplicate runners inside one process.

Configuration Surface

Open a store with:

RocksDBDataStore.open(
    keepAllVersions = true,
    relativePath = "path/to/db",
    dataModelsById = mapOf(1u to Account, 2u to Course),
    rocksDBOptions = DBOptions(),
    migrationConfiguration = MigrationConfiguration(
        migrationHandler = { context -> /* ... */ MigrationOutcome.Success }
    ),
    versionUpdateHandler = { store, stored, new -> /* ... */ }
)

Notes:

• keepAllVersions controls whether historic families are created and maintained.
• relativePath points to a directory where the store manages all column families.
• dataModelsById determines the model ids used in column family names and writes.
• Pass custom DBOptions and ColumnFamilyOptions via the datastore open path if you need to tune cache, compaction, bloom filters, etc.

Testing

The store/test module contains a reusable test suite that the RocksDB store runs in commonTest. It covers adds/changes/deletes, uniques, indexes, scans, and historic queries. See RocksDBDataStoreTest.kt and related tests in src/commonTest.

File Map (selected)

• RocksDBDataStore.kt – open/init, CF creation, request actor, index backfill, close.
• TableType.kt – column family naming (type byte + varint id).
• BasicTableColumnFamilies.kt, TableColumnFamilies.kt, HistoricTableColumnFamilies.kt – handles per model.
• processors/* – per-request processing and scan logic.
• processors/helpers/* – version bytes, get/set value/index/unique, soft delete checks.
• Transaction.kt, DBAccessor.kt, DBIterator.kt – local transaction and read wrappers.

Complete-value checks and empty collection removal

Check statements decode scalar qualifiers directly and materialize complex values once from the current native transaction. A missing expected scalar fails validation; collection counts are never compared to whole collection values. Checks after a change see the staged writes, and validation failure rolls back the record savepoint. Root list/set/map counts have no type indicator: a zero-count byte means a present, empty collection. Nullable removal deletes that qualifier and its descendants, including the empty case, and retains the normal history tombstones.

Physical backup and recovery

RocksDBBackupRepository wraps the engine backup APIs in serialized IO operations. It embeds MARYK_META.yml and history settings in each backup’s application metadata, prevents overlap with database directories, and restores only into new directories. A store lifecycle guard keeps native resources alive during backup; schema/index migrations must finish before capture. See backup and restore for how physical recovery complements the shared portable record backup format.