Architecting High-Throughput Durable Execution Engines in 2026: Deterministic Coroutine State Checkpointing, Raft-Replicated Virtual Actor Logs, and Zero-Copy Micro-Wasm Sandboxing

Key Takeaways
- Zero-copy Micro-Wasm sandboxing provides efficient, deterministic runtime isolation for workflow execution.
- Deterministic coroutine $O(1)$ delta checkpointing replaces $O(N)$ event history replay for state durability.
- Multi-Raft consensus replicates virtual actor logs, ensuring durable state and high availability.
- New engines overcome $O(N)$ replay penalties and database IOPS saturation of prior designs.
- The paradigm shifts to incremental snapshots of the workflow's executing memory image as the durable entity.
A durable execution engine lets a long-running workflow survive process crashes, restarts and network partitions without hand-written recovery code. The first generation did this by replaying an event history from line one of the workflow code. A faster design combines coroutine or memory-image checkpoints, a replicated per-workflow log, and a sandbox that makes execution reproducible. This guide explains how those pieces fit together, what each one costs, and where the common claims need caution.
Why Event-History Replay Hits a Ceiling
Engines such as Temporal and Cadence persist every workflow decision as an event in an append-only store. When a worker restarts or evicts a workflow from its cache, it reloads the history and re-executes the code, with recorded results substituted for side effects. This is robust, but it carries three structural costs:
- Replay cost grows with history length. Rebuilding state means processing every event again. Mature engines blunt this with sticky worker caches and "continue-as-new" (restarting a workflow with a fresh history), but the underlying cost remains for cache misses and failovers.
- The store becomes the bottleneck. If thousands of workflows wake at once, for example from a batch timer, every worker reads history from the same database.
- Code changes can break determinism. Inserting a step into a workflow that has in-flight executions makes replay diverge from the recorded history. Versioning APIs or patch markers are required to avoid this.
A different family of engines avoids full replay. Restate journals each step and keeps state close to the compute, while Azure Durable Functions uses event sourcing with replay. Both show the trade-off space is wide. Treat any single set of latency figures as workload-dependent; measure your own.
Core Architecture: Four Layers
A high-throughput design usually has four layers, each addressing one of the costs above.
- Partition-aware ingress. Requests are routed by workflow ID through consistent hashing so one owner handles each workflow at a time.
- Deterministic execution sandbox. User code runs where every source of non-determinism is intercepted.
- Checkpointing and a per-partition log. Step results and state deltas are committed durably before the engine acts on them.
- Replication and compaction. The log is replicated for availability and truncated once a checkpoint supersedes it.
The comparison below summarises the design choices.
| Aspect | Event-history replay | Checkpoint plus replicated log |
|---|---|---|
| Recovery work | Proportional to history length | Load latest checkpoint, replay short log tail |
| Storage | Central database, append-only | Partitioned log, compacted periodically |
| Failover | Reload history, re-execute code | Promote a replica that already holds the log |
| Main risk | Replay cost and version drift | Snapshot size, snapshot format compatibility |
| Operational weight | Mature, well-documented | More custom engineering unless you adopt a product |
Note that checkpointing does not remove the need for determinism. Recovery from a snapshot plus a log tail still requires that re-applying the tail produces the same state.
Deterministic Coroutine Checkpointing
What determinism requires
A workflow is deterministic when the same inputs always produce the same sequence of decisions. Real code breaks this by reading the clock, generating random numbers, iterating unordered maps, or calling the network directly. The runtime must trap each of these at a boundary:
- Time: return a logical timestamp recorded in the log, not the wall clock.
- Randomness: seed a PRNG from a value committed to the log, so replays draw the same sequence.
- I/O: turn every external call into a recorded intent. The coroutine suspends, the host performs the call, and the result is journaled before the coroutine resumes.
Snapshots versus journals
There are two practical ways to make a suspended coroutine durable. A journal records each step's inputs and outputs and rebuilds state by re-running code against them. A memory snapshot stores the sandbox's memory image, so resuming means restoring pages instead of re-executing logic. Snapshots can make resume cost roughly independent of history length, but they bind you to the runtime's memory layout. Upgrading the language runtime or the workflow code can invalidate old snapshots, so keep a journal fallback or a migration path.
Dirty-page tracking
Writing the full memory image at every suspension point is wasteful. The usual optimisation is to capture only modified pages. In a WebAssembly runtime you control the linear memory, so you can track writes in a bitmap and copy only flagged pages at checkpoint time. On native processes, the equivalent mechanisms are write-protection faults, userfaultfd on Linux, or soft-dirty bits exposed through /proc. Cost then scales with the amount of memory changed per step rather than total heap size, though very write-heavy steps can still produce large deltas.
The sketch below shows the core idea in a few lines:
<pre><code>const PAGE: usize = 65536; // Wasm page size fn mark_dirty(bitmap: &mut [u64], offset: usize) { let page = offset / PAGE; bitmap[page / 64] |= 1u64 << (page % 64); } // At checkpoint: copy only pages whose bit is set, then clear the bitmap.</code></pre>
A production implementation must also handle bulk memory operations, memory growth and host-side writes into guest memory, all of which can modify pages without passing through your tracked write path.
Raft-Replicated Virtual Actor Logs
A virtual actor is an addressable stateful object that the runtime activates on demand, keeps in memory while busy, and passivates when idle. Pairing each workflow with a replicated log gives you durability without a central database on the hot path.
How commits work
- The actor's owner receives an event or timer.
- The step runs and produces a state delta plus outbound command intents.
- The delta and intents are appended to the partition's Raft log.
- The leader waits for a quorum acknowledgement, then releases the intents for execution.
Committing before acting is the key rule: an external call must never be issued unless its intent is durable, otherwise a crash can lose or duplicate it.
Multi-Raft and its limits
Running one Raft group per partition (Multi-Raft, as used by CockroachDB and TiKV) spreads leadership across nodes and bounds the blast radius of a failure. Each commit costs at least a network round trip to a quorum plus a durable write, so latency is governed by your fsync and network characteristics, not by the consensus algorithm alone. Sub-millisecond commit latency is realistic only inside one availability zone with fast local storage; cross-zone or cross-region quorums will be slower. Verify against your own hardware rather than a published number.
Compaction
Logs grow without bound unless truncated. After a checkpoint is durably replicated, entries older than it can be discarded or moved to cheaper tiered storage. Snapshot installation for lagging followers uses the same mechanism, so checkpoint size directly affects recovery time for a slow replica.
Wasm Sandboxing for Isolation and Reproducibility
WebAssembly suits this problem for three reasons: it has a small, well-defined instruction set; the host controls every import, so non-determinism can be removed at the boundary; and linear memory gives a contiguous region that is simple to snapshot.
- Isolation: each workflow instance gets its own linear memory with a hard limit, plus fuel or epoch-based interruption to stop runaway loops.
- Determinism: disable or normalise non-deterministic features. Floating-point NaN bit patterns, threads, and relaxed SIMD instructions can differ across hosts, so pin the engine version and configuration.
- Start-up cost: runtimes such as Wasmtime support pre-instantiation and pooled allocation, which makes instance creation very cheap. Microsecond-scale figures are plausible with pooling, but the number depends on module size and configuration, so benchmark your own modules.
"Zero-copy" claims deserve scrutiny. Memory can be mapped or shared between host and guest to avoid copying large payloads, but checkpoints still have to be written out to the log, so some copying is unavoidable at the durability boundary.
Build or Adopt: A Decision Guide
Building an engine of this kind is a multi-year undertaking. Before you commit, work through these questions:
- Do existing engines (Temporal, Restate, Azure Durable Functions, AWS Step Functions) meet your throughput and latency needs when properly tuned?
- Do you need to run untrusted or multi-tenant code, where Wasm isolation changes the security model?
- Can your team operate a consensus system, including membership changes, snapshots and disaster recovery?
- Is the workflow language fixed, so a snapshot-compatible runtime is feasible?
For most teams, adopting an engine and applying its scaling features (sticky caches, continue-as-new, sharding, rate limits) is the better first move. For related design thinking, see Microservices Architecture Guide.
Common Mistakes and Pitfalls
- Acting before committing. Issuing an external call before its intent is durable breaks exactly-once reasoning.
- Assuming "exactly-once" is free. External side effects are at-least-once at the boundary; make them idempotent with stable idempotency keys.
- Ignoring snapshot compatibility. A runtime or code upgrade can make existing snapshots unreadable. Version the snapshot format and test restores across versions.
- Unbounded workflow state. Large histories or heaps defeat checkpointing; split long-lived work into child workflows or restart with fresh state.
- Trusting benchmark headlines. Latency and throughput depend on storage, network topology, payload size and step granularity.
- Leaving non-determinism unpinned. Map iteration order, floating-point behaviour and library upgrades all cause replay divergence.
Key takeaways
- Replay-based engines are reliable but pay a recovery cost that grows with history; mitigate it first with caching and continue-as-new.
- Snapshot-based resume can make recovery independent of history length, at the price of coupling to the runtime's memory format.
- Commit intents to a replicated log before performing any external action, and make external effects idempotent.
- Wasm gives controllable isolation and a snapshot-friendly memory model, but determinism still has to be enforced deliberately.
- Treat published latency figures as hardware- and workload-specific, and benchmark your own scenario.
- Adopt an existing engine unless you have a clear requirement it cannot meet.









