Maryk + FoundationDB: Architecture Overview

This document describes how the FoundationDB storage engine is structured inside Maryk, how requests flow through the system, and how the main components relate. It is intended for contributors who want to understand and maintain the implementation.

Big Picture

Maryk defines a storage‑engine‑independent core. Each engine implements the same high‑level operations (Add/Change/Delete/Get/Scan/GetChanges/ScanChanges/Updates). The FoundationDB engine maps those operations onto FoundationDB transactions, directories (subspaces), and range reads.

At runtime:

• FoundationDBDataStore owns the FDB client, opens the directory tree, and runs a single‑threaded coroutine actor that processes requests sequentially.
• “Processors” map each request type to concrete reads/writes against FDB.
• “Helpers” encapsulate encoding/decoding, qualifier matching, version math, and index/unique management.
• An in‑memory Cache helps de‑duplicate value decoding when a request needs the same property multiple times (e.g. aggregations).

Capabilities in the FDB engine:

• keepAllVersions is fully supported for data, uniques, and indexes. Latest and historic index scans are supported; historic scanning is used when a toVersion is provided.

Core Components

FoundationDBDataStore

• Entry point to the engine.
• Constructor flags:
  • keepAllVersions: whether to maintain historic data alongside latest.
  • fdbClusterFilePath, directoryRootPath: FDB connectivity & subspace root.
  • clusterUpdateLogConfiguration: optional cluster-wide live update propagation for executeFlow (writes updates into an FDB log and tails them back into the in-memory listener flow). Models with sensitive properties are rejected when this is enabled because cluster-log payload encryption is not implemented. Upgrading its legacy HLC-ordered log is coordinated: quiesce all readers and writers, upgrade every binary, then restart. Consumers drain persisted legacy entries before storing a one-way commit-ordered cursor. Mixed old/new binaries and rollback after activation are unsupported because old binaries cannot enforce or understand the transition.
• Initializes per‑model directories via DirectoryLayer.
• Launches the store actor (coroutine) to process incoming requests one by one. Add, change, and delete processors use one FDB transaction per object; multi-object requests can therefore return partial results. Reads use transactions or iterators scoped to the required operation.

Process lifecycle

FoundationDBDataStore.close() releases per-store resources while leaving the process-wide FoundationDB client network available to other stores. A Native host must close all stores, then call FDB.instance().shutdown() during process termination. Shutdown stops the shared network worker before native runtime teardown and is terminal: the network cannot restart in that process.

Public atomic requests and transaction scopes

Explicit atomic requests and serializable transaction journals use one native transaction per replay attempt across participating models, schema fences and receipts. Preview aborts its native transaction; application suspension retains no native snapshot. A datastore revision fence detects intervening Maryk writes, including writes outside the models selected by the scope. A replay with an expected revision reads this fence with a conflict dependency. Requests without an expected revision read the token through snapshot() and rely on their native record/range conflicts, avoiding unrelated retries without weakening suspended-scope validation. Atomic requests keep initial-state checks; interactive journals read earlier staged writes and support repeated mutations and hard deletion. See the shared transaction contract.

Cluster Update Log (Optional)

Goal: cluster-wide executeFlow updates (multi-writer, multi-reader) using only FoundationDB.

Subspaces (under the configured store root):

• __updates__/v1/log: append-only log entries written in the same transaction as the table mutation.
  • Current keys order by commit: (shard, modelId, versionstamp, hlc_be_8, keyBytes, originIdBytes). Retained legacy keys use (shard, modelId, hlc_be_8, keyBytes, originIdBytes, versionstamp).
  • Value contains (originId, modelId, type, version, keyBytes, payload) where payload is values (Add), changes (Change), or hardDelete flag (Delete).
  • Readers validate the complete six-field stored identity and require its model, event HLC, record key and origin to match the decoded payload in both formats. Malformed or mismatched entries are skipped while the physical cursor advances; stored rows remain untouched. Chunked payloads use the same check after reconstruction. Commit versions come from the validated versionstamp, independently of the event HLC.
• __updates__/v1/consumers/<consumerId>: per-consumer cursors, one per shard. Each node/process must use a unique consumerId.
• __updates__/v1/heads: “wake keys” updated with versionstamped values. Tailers watch these to avoid polling the entire log when idle.
• __updates__/v1/hlc: per-node HLC markers. Each writer updates hlc/<consumerId> (8 bytes big-endian HLC timestamp) on every append.
• __updates__/v1/hlc_max: shard-local cluster HLC max markers. Each writer updates hlc_max/<shard> via atomic BYTE_MAX on every append.
  • A background HLC syncer on every node watches heads and refreshes max(hlc_max/*, hlc/*) so local write versions never lag behind cluster HLC, even with zero active executeFlow listeners.

Sensitive Field Encryption (Optional)

• Property wrappers can be marked sensitive in model definitions (sensitive = true).
• FoundationDB store encrypts sensitive value payloads before writing table/historic entries.
• Read paths auto-decrypt when encrypted payload marker is present.
• Current constraints:
  • only simple value properties are supported
  • sensitive+unique uses deterministic lookup tokens (provider must implement SensitiveIndexTokenProvider)
  • sensitive+indexed is not supported

Semantics:

• Delivery is at-least-once. On failure/restart (or on consumerId change), consumers can see duplicate updates within the retention window.
• No strict global order across shards. Ordering is stable within a shard (HLC + versionstamp).
• Retention is time-based. A background GC clears per-shard ranges older than a cutoff (retention + skew margin) using HLC order.

TableDirectories

Encapsulates the FDB subspaces per model:

• Generic: meta
• Non‑historic: keys, table, unique, index.
• Historic: adds table_versioned, unique_versioned, index_versioned.

This provides simple, fast pack() prefixes for building FDB keys.

Processors (per request)

• processAddRequest → validates input, checks uniques, writes (version || value) into table, updates indexes/uniques, and marks creation/latest versions.
• processChangeRequest → reads current values, validates, applies modifications, writes new (version || value) and updates indexes/uniques (including historic tombstones/snapshots if enabled).
• processDeleteRequest → soft delete writes a tombstone; hard delete clears keys, table, and optionally historic. Updates indexes/uniques accordingly.
• processUpdateResponse → applies externally supplied updates (Addition/Change/Removal/InitialChanges) into the store to synchronize state; InitialValues and OrderedKeys are rejected as they do not contain sufficient version/change context.
• processGetRequest → fetches values for keys, obeying toVersion, select, filters, and filterSoftDeleted.
• processScanRequest → scans by key or index (based on order), performs filtering, and returns values (or aggregates) for up to limit rows.
• processGetChangesRequest/processScanChangesRequest → same navigation as Get/Scan but reading “VersionedChanges” instead of full values, taking fromVersion/toVersion/maxVersions into account and returning the sortingKey (for index scans).

Each processor uses a combination of:

• A small transactional body (tc.run { tr -> … }) for reads/writes.
• Iterators (getRange(...).iterator()) for range scans on keys/table/index.
• The shared value codecs and qualifier matchers from core.

Note on index‑based Changes: the returned sortingKey is the (indexValue || key) byte slice (without version). Clients can use it for stable pagination and correlation across engines.

Helpers

Some representative helpers used across processors:

• packKey(prefix, vararg segments) builds full FDB keys from subspace + dynamic parts.
• setValue, setLatestVersion, setCreatedVersion write latest and creation timestamps + values, and mirror into historic subspaces when enabled.
• toReversedVersionBytes, readVersionBytes, readReversedVersionBytes for version encoding/decoding.
• Qualifier matching utilities to evaluate Maryk filters against stored bytes efficiently (direct gets for exact references, small range scans for fuzzy references).
• Index/unique writers and historic value writers.
• Zero‑free qualifier encoding: encodeZeroFreeUsing01/decodeZeroFreeUsing01 ensure qualifiers in historic keys contain no 0x00 bytes. A single 0x00 separator is placed between the encoded qualifier and the inverted version bytes for unambiguous split and ordering; forward scans then yield latest‑first entries efficiently.

Legacy stores can retain a current soft-delete marker without its matching historic row. A current marker at or before the requested snapshot determines the deletion state, even when an older historic marker exists. Later markers cannot affect earlier snapshots. Historic values reconcile this state after decoding, so projections and soft-delete filters agree. Change readers merge only in-bound transitions and keep their soft-delete maxVersions limit; finite history uses the exact event version and can recover a restoration even when filtering deleted records.

Request Flow (Example)

Add

1. Actor receives AddRequest with N objects.
2. For each object:
   • Validate object; compute (key, version).
   • Read unique keys to ensure no conflicts; throw validation if found.
   • Write (version || value) per property and mark creation/latest.
   • Write index/unique entries.
3. Emit updates on the shared flow (for listeners).

Get

1. Actor receives GetRequest(keys, select, where, toVersion, aggregations, …).
2. For each key inside a transaction:
   • Check existence (creation version) and run filter/soft‑delete checks.
   • Read either latest from table or historic from table_versioned up to toVersion.
   • Apply select graph to avoid unnecessary reads.
3. Aggregate fields if requested, using Cache to avoid re‑decoding the same value twice.

Scan

1. Compute key ranges and/or index ranges (depending on order).
2. Scan keys (table scan) or index (index scan), filter rows, and collect up to limit.
3. Return FetchByTableScan or FetchByIndexScan metadata (direction, start/stop keys, etc.).

GetChanges / ScanChanges

1. Navigate like Get/Scan, but read values as “versioned changes” rather than current snapshots.
2. For latest only, read from table and attach versions; for historic or maxVersions > 1, read from table_versioned and stop at maxVersions per field.
3. Return sortingKey for index scans (the (indexValue || key) bytes) so consumers can page or correlate properly.

Versioning Model

• “Latest” and “Historic” are physically separate, so point reads are small and “as of” queries are clean.
• Versions use Maryk’s HLC (a 64‑bit timestamp) with inverted byte encoding in historic keys to make newest entries come first in range scans.
• Concurrency is handled by FDB transactions; we read‑for‑write inside the same transaction for uniques/validations.

Filtering

Filters compose (And/Or/Not) around primitives (Exists, Equals, Range, Regex, ValueIn, Prefix…). The engine evaluates them by matching property references against stored qualifiers:

• Exact: direct value lookup (fast path).
• Fuzzy: small range scan under the property’s qualifier prefix.

Soft deletes are enforced as an independent check.

The FDB engine supports fuzzy qualifier and sub-reference filtering. Complex scans should still prefer secondary indexes for predictable performance.

Indexes and Uniques

• Indexes: indexRef + (indexValueBytes || keyBytes) → version (latest) or … + inv(version) → tombstone/snapshot (historic).
• Uniques: (uniqueRef || valueBytes) → (version || keyBytes) (latest) and mirrored into historic if enabled.

For scans, we slice by (indexValue || key) to include/exclude startKey and page in either direction.

Error Handling and Migrations

Full migration operations guide: Migrations.

• Any RequestException or validation error is wrapped in structured statuses (e.g. AlreadyExists, ValidationFail).
• Schema migrations are coordinated through the model subspace.
• migrationConfiguration.migrationHandler uses MigrationContext and returns MigrationOutcome (Success, Partial, Retry, Fatal).
• migrationConfiguration = MigrationConfiguration(...) from maryk.core.models.migration.
• Optional phase hooks also live there: migrationExpandHandler, migrationVerifyHandler, migrationContractHandler.
• Long migrations can hand off from startup to background (migrationConfiguration.migrationStartupBudgetMs, migrationConfiguration.continueMigrationsInBackground).
• Pending models are request-blocked until migration completes (assertModelReady gate).
• Runtime operations:
  • pendingMigrations, migrationStatus, migrationStatuses, awaitMigration
  • pauseMigration, resumeMigration, cancelMigration
• Default lease is distributed (FoundationDBMigrationLease):
  • per-model lease record in metadata
  • owner token + TTL (migrationLeaseConfiguration.migrationLeaseTimeoutMs)
  • heartbeat renewal (migrationLeaseConfiguration.migrationLeaseHeartbeatMs)
  • takeover after TTL expiry when owner is gone

Why this design?

• FoundationDB excels at ordered keyspaces and range reads. Separating current vs historic and composing keys as (context prefix || logical key || qualifier || version) plays to these strengths.
• The engine mirrors the Maryk semantics already proven in other backends (RocksDB), which keeps the behavior consistent across storage engines and simplifies test reuse.

Where to look in code

• Data store: store/foundationdb/src/commonMain/kotlin/maryk/datastore/foundationdb/FoundationDBDataStore.kt
• Processors: store/foundationdb/src/commonMain/kotlin/maryk/datastore/foundationdb/processors/…
• Helpers: store/foundationdb/src/commonMain/kotlin/maryk/datastore/foundationdb/processors/helpers/…
• Tests: store/foundationdb/src/commonTest/kotlin/maryk/datastore/foundationdb/FoundationDBDataStoreTest.kt and shared test suite in store/test/…

Complete-value checks

Both check-only and mixed Change requests compare complete values from the current native transaction. Scalars use direct qualifier reads; each Check with complex properties materializes the record once. Collection counts are not decoded as collection values. A missing expected property fails validation. Checks after a change see preceding staged writes, and a failed check aborts the record mutation.

Native API read and write optimizations

Clock watermark buckets use atomic BYTE_MAX on canonical eight-byte unsigned big-endian HLC values. This keeps the maximum across concurrent writers without reading the bucket in each mutation. This internal metadata must retain its canonical encoding; corrupt watermark metadata requires recovery rather than using normal writes as a repair mechanism.

Main table and index scans start with at most the requested row count and grow bounded candidate batches up to 512 when more candidates are needed. Batches use the binding’s default streaming mode; Java asList already optimizes finite limits, and the local probe found no benefit from forcing WANT_ALL. Current table scans reuse the creation version in their key row. Current index scans prefetch candidate creation markers before filtering, and cluster-log payloads fetch up to eight chunks together. Outstanding futures are canceled when a batch completes or fails; decoding and results preserve storage order and the request’s pinned read version.

Request/maintenance policies and per-attempt tracing hooks are described in the configuration guide.

Retry markers and asynchronous ownership

Ordinary native write attempts share one invocation ID and retained outcome, created outside the retry callback. Private markers live in __meta__/ordinary_receipts, separate from durable serialized atomic receipts and their format conversion. Successful invocation cleanup uses a bounded queue of unique invocation keys rather than copying a growing set for each acknowledgement. Failed cleanup batches are restored ahead of newer keys within the existing 65,536-key cap; clearing replayed keys is idempotent. Abandoned markers require quiescent maintenance if they accumulate after crashes. Markers do not provide cross-call idempotence without an application-supplied stable transaction ID.

Suspending transaction callbacks run in a supervisor scope so one failed attempt cannot cancel its parent before FoundationDB retries it. A canceled operation cancels its native transactions as well as its futures. Prepared range readers use the common Flow adapter to own iterator/future cleanup, and enforce atomic byte budgets before collecting rows. Ordinary reads, transaction previews/commits and restores all use the configured request limit. A native attempt shares one request budget across its members; the next attempt receives a fresh budget. Each decoded Get record releases its raw buffers immediately, while at most eight record reads remain prefetched. Get decoding uses already-awaited rows; synchronous codec/filter callbacks keep a compatibility bridge rather than introducing a second coroutine inside every value callback.

Cluster head values are compared with the previous idle pass before committed watches are armed, covering the scan/watch gap. Separate HLC and tailer baselines avoid consuming each other’s signals. A local conflated listener signal wakes the tailer when subscriptions change. Watch cancellation propagates through the JVM mapped future to its native source; native awaiting-coroutine cancellation also cancels the underlying future.

Maintenance split points are derived from data/history bytes and normalized to whole record keys. Workers keep the same schema fence, independent scratch record namespaces and bounded native transactions. Final schema publication waits until all workers complete; cancellation leaves the rebuild fenced and recoverable.

Reproducing performance measurements

Run :store:foundationdb:runFoundationDBPerformanceProbe against the managed isolated local cluster. It verifies returned rows/statuses and reports median and p95 range timings, scan-versus-point ownership lookup, and bulk import with one versus eight concurrent transactions. Timings are diagnostic output, not unit-test assertions or guarantees for a remote production cluster.

Before publishing binding changes, use -PfoundationDbBindingsPath=/absolute/path/to/foundationdb-multiplatform to compile that checkout with Maryk’s verified toolchain. This local source override is explicit; normal builds resolve the pinned published binding artifact. Native focused checks also accept -PfoundationDbNativeTestFilter=package.Class.*.

Read allocation and receipt cleanup comparisons are also available through :store:foundationdb:runFoundationDBPerformanceProbe --args=reads and --args=receipts. The receipt probe retains the previous set-copy algorithm as a control. Both modes check their results, warm up, and report allocation counters when the JVM exposes them. JVM and Native release measurements should be kept separate from debug-test timings.

Stores without sensitive fields pass payload slices directly to the decoder; sensitive stores retain context-bound decryption. Canonical strings decode from bounded byte arrays without rebuilding a streaming buffer. Other definitions and noncanonical markers use the general decoder. Persisted encodings and read budgets are unchanged.

Optional JVM direct-buffer queries

The binding’s JVM runtime extensions can reduce temporary heap allocation during range reads. Configure them once at application startup, because the SDK setting and buffer pool are process-wide:

import maryk.foundationdb.ApiVersion
import maryk.foundationdb.FDB
import maryk.foundationdb.jvm.enableDirectBufferQuery
import maryk.foundationdb.jvm.resizeDirectBufferPool

val fdb = FDB.selectAPIVersion(ApiVersion.LATEST)
fdb.resizeDirectBufferPool(poolSize = 8, bufferSize = 512 * 1024)
fdb.enableDirectBufferQuery(true)

This is opt-in: eight buffers reserve about 4 MiB of native memory, and the SDK initializes its 64 MiB default pool before resizing. Larger/concurrent queries need their own measurements and sizing. Native clients do not use this Java pool. Maryk leaves the SDK default disabled; lower heap allocation does not by itself imply lower total memory use or a useful latency improvement.