Hierarchical multitarget error diagnostic system for zkOS

First, what are we working with, what is that ZKsync OS really? Here is a quick overview, assuming little exposure to the blockchain world.

A typical smart contract for Ethereum is written in Solidity, compiled into the bytecode for Ethereum Virtual Machine (EVM), then executed. EVM runs on servers which back the Ethereum network. Once in every few seconds some nodes in this network are randomly selected and they have to agree on the execution results through a consensus mechanism. The blockchain state is eventually updated to reflect these results.

The extensions to Ethereum, the rollups, are offloading execution of smart contracts to separate networks, then publishing the results in bulk on the Ethereum blockchain. Some of them, the ZK-rollups, can publish a small difference between states along with a proof, that the execution producing this state was correct and fair. The proof is backed by cryptography guaranteeing that no one could forge it and publish false execution results.

In case of ZKSync, the underlying execution platform is not strictly EVM, but a virtual machine conforming to the RISC-V architecture. EVM interpreter sits on top of RISC-V virtual machine, compiled into native code for RISC-V. We can add other execution environments, running smart contracts compiled into WASM, native RISC-V instructions and so on. These contracts could, in theory, also interact with each other.

Then ZKsync Airbender generates proofs of correctness for RISC-V execution, not for EVM itself. It is a stronger proof so it also guarantees that everything executed on top of RISC-V was proper.

ZKsync OS is an implementation for a state transition function that enables multiple execution environments (EVM, EraVM, Wasm, etc.) to operate within a unified ecosystem. Think of it as an operating system, a resource manager, controlling execution environments, feeding them transactions, collecting results, managing interop, and so on. To execute a smart contract, we launch the virtual RISC-V machine with an EVM interpreter on top of it; there is some windup when the interpreter is initialized and accepts inputs. All of it becomes a part of execution trace for which Airbender generates a proof.

ZKsync OS compiles to three distinct target profiles, each with different requirements:

For proving, ZKsync OS compiles to a minimal RISC-V binary for zero-knowledge proof circuits. There, you pay "gas" for every instruction. Yet the same code must provide rich, actionable diagnostics when run natively for development and debugging. The forward run profile sits in the middle: we are fine with a slight overhead and we can optimize for an average case, where performance will not be substantially affected.

I started working on ZKSync OS when it has already been several months in development. The system was growing organically, there were some leftovers from earlier iterations, and the codebase was moving fast. It was also a critical piece of software – a dangerous combination, promoting brittleness and ad-hoc solutions. We have already started to suffer from inconsistencies between how errors were handled in different parts of the system: there were multiple error hierarchies here and there, some of them redundant, some of them incompatible.

It was the right moment to intervene, clean out the debris, refine the emerging architecture, and imbue it with error diagnostics. My intention was to:

• Structure and enforce subsystem boundaries.
• Explicitly describe possible interface violations.
• Adapt to tradeoffs between runtime costs of diagnostics and its completeness.
• For debugging, provide rich context with ZKSync OS state, precise error location, and the propagation path.
• Force good practices on developers. New errors should not be defined spontaneously, they should be properly described and categorized. Subsystem interface violation, a bug in a subsystem, resource exhaustion correspond to different responsible parties and recovery strategies.
• Automate error propagation.
• Maximize static guarantees from Rust. At the same time the added infrastructure should not become too complex (and hard to maintain), and making use of unstable features of type system could break the codebase in the future.
• compatible with nostd and noalloc in RISC-V builds, but may use them in forward run builds.

With this in mind we also have to target three platforms with different sets of constraints and stakeholders:

The verbosity level should be a compile-time decision – in the proving binary, we need to eliminate all debugging overhead.

In addition to that, there was already an ad hoc solution – a bunch of enums scattered across the codebase, not forming a single hierarchy, without defined propagation paths. But we were used to them, and forcing our colleagues into a new way of doing things adds friction, so we needed a clear, gentle migration path. We could not switch all subsystems to the new error hierarchy at once because it would put too much strain on the codebase, on reviewers and on other developers. Still, it had to be done fast, because the development moves rapidly.

Lastly, such error system would introduce additional complexity in order to reduce it on use site. I had to make sure there are some diagnostic tools for the error system itself so that developers could pick it up quickly and be guided in case they misuse it.

I want to approach the design problem as a system engineer, so in case you are not familiar with how system engineering views systems, errors, stakeholders, here is a refresher.

Who cares about knowing what error occurred in ZKSync OS? It concerns many parties, some human, some organizations, some code. I separate them because their requirements for diagnostics differ. Humans, for example, want legible and actionable error messages; code cares about errors that are easy to parse/handle. Humans won't be as much puzzled by a change in error message, or error code, but tooling will likely grow dependent on them¹.

• Smart contracts, run on top of EVM controlled by ZKSync OS
  • and their developers,
  • and the development tooling for both native Ethereum and EVM-based rollups
    • which includes environments and nodes like Anvil, Foundry, Hardhat…
    • … and compilers for Solidity – solc or our new LLVM-based Solidity compiler solx…
    • … and compilers for other languages like Vyper,
    • … and the IDEs like Visual Studio Code,
    • … and the language plugins for editors like;
    • and the developers of this tooling;
• Developers of ZKSync OS itself
  • … including developers of individual subsystems,
  • … and developers who own the subsystem differ from developers who just use it through its interface.
• Proof generator
  • … and its developers;
• Sequencer (which directly interfaces with ZKSync OS)
  • … and its developers;
• Security engineers
• Auditors

From 2025 on, the developers category includes both humans and AI agents. They act on errors differently:

• a human sees an error, builds a mental model, maybe checks docs, then edits code;
• an LLM agent gets the error as a text token stream, pattern-matches against training and context, and immediately emits an action.

Humans arguably tolerate message changes better than agents – we can always check the documentation by the error code. Agents may get confused by rewording. Humans like terser, compressed messages that fit our attention span; agents prefer longer messages with richer context.

Even the simplest little systems have surprisingly many direct and indirect stakeholders. Here is a hypothetical: 1) we make a platform for smart contract execution that has unparalleled diagnostics 2) debugging contracts becomes so easy it becomes a competitive advantage. Now other execution platforms (using custom ZK VMs, on top of other blockchains like Solana or Tezos) are concerned. First, they might need to keep up and improve their diagnostics experience. If we gained enough traction, both developers and tooling become dependent on our formats, codes, conventions, usage patterns and so on. This might become a collaboration opportunity – joining forces and working on a unified debugging experience, which would improve life for everyone.

Finally, we are not frozen in time – stakeholders may change, the requirements may evolve.

Most stakeholders see the system from one of three sides:

1. Subsystem developers see the internals of ZKSync OS, but they do not possess the whole knowledge of the system. They typically observe the system at subsystems boundaries.
2. Developers of prover, sequencer and tooling may observe ZKSync OS at its outermost boundary, where it accepts transactions and returns results.
3. Developers of smart contracts interface with other systems, for example sequencer. Errors from ZKSync OS may be wrapped and cascaded to them.

The origin of error defines how much information does a stakeholder need to make sense of it. So this will be a starting point for our error taxonomy. We can start classifying errors based on their origin:

• Defects, or internal errors, which should be fixed at origin.
• Interface misuse, when we violate the protocol of interaction with a subsystem. It can be fixed by those who initiated interaction.
• Runtime errors, for which, as we said, the origin is more loosely related to the cause.

Having three kinds of errors – defects, contract violations, and runtime errors – we start working them into the system.

The first step is to solidify the system structure. It means putting defined boundaries around subsystems, so that for interface errors their subsystem is clear.

ZKSync OS is written in Rust, which provides structure through language level modules. But these modules are structural; they do not precisely map to the functional architecture of the system.

By convention, we will contain every subsystem within a branch of module tree. When a subsystem operates, errors can appear. Every subsystem boundary has a declared error type enumerating the errors permitted to cross that boundary.

Our task is then to explicitly draw boundaries around subsystems and make their interfaces use the corresponding error types. Here is my solution:

// Collects all static information about a subsystem
pub trait Subsystem: core::fmt::Debug {
    const SUBSYSTEM_NAME: &'static str;
    // Enum describing all possible interface violations
    type Interface: InterfaceErrorKind; // implement InterfaceErrorKind for an interface type error of a subsystem
}
pub enum SubsystemError<S: Subsystem> {
    LeafUsage(InterfaceError<S::Interface>),
    LeafDefect(InternalError),
    LeafRuntime(RuntimeError),
}

Only interface misuse is specific to a subsystem. A defect, or a runtime error can be passed around freely, wherever they originated from.

We only missed one option – when one these three kinds of errors was raised in a subsystem deeper, then cascaded to the interface for this subsystem. Then we wrap the error and pass it up the call stack. Here is an upgrade:

pub trait Subsystem: core::fmt::Debug {
    const SUBSYSTEM_NAME: &'static str;
    type Interface: InterfaceErrorKind;
    // Enum describing all possible wrapped errors
    type Cascaded: ICascadedInner;
}
pub enum SubsystemError<S: Subsystem> {
    LeafUsage(InterfaceError<S::Interface>),
    LeafDefect(InternalError),
    LeafRuntime(RuntimeError),
    Cascaded(CascadedError<S::Cascaded>),
}

Now all the error propagation paths are defined statically. Here is a simplified example of such description:

trait BootloaderSubsystem {
    const SUBSYSTEM_NAME: &'static str = "Bootloader";
    type Interface: InterfaceErrorKind = BootloaderInterfaceError;
    // Enum describing all possible wrapped errors
    // Implement ICascadedError for an enum collecting all inner subsystem from
    // which an error can reach the bootloader boundary
    type Cascaded: ICascadedInner = BootloaderWrappedError; 

}
impl InterfaceErrorKind for BootloaderInterfaceError { ... }
enum BootloaderInterfaceError {
       CantPayRefundOverflow,
       TopLevelInsufficientBalance { balance: u256 },
} 
impl ICascadedInner for BootloaderInterfaceError { ... }
enum BootloaderWrappedError {
       Balance(BalanceSubsystemError),
       EEError(EESubsystemError),
       Nonce(NonceSubsystemError),
}

With this code, we are able to use SubsystemError<BootloaderSubsystem> as a return type for the functions that make up an interface for a subsystem, like that

fn some_bootloader_function(
    &mut self,
    from_ee: ExecutionEnvironmentType,
    ...
) -> Result<(), SubsystemError<BootloaderSubsystem>> { // or make an alias like BootloaderError
    ...

What of subsystems that have no interface errors (very simple) or no cascaded errors (leaf subsystems)? We can assign an empty enum to the Interface and Cascaded types of Subsystem trait:

pub enum NoErrors {}
pub trait Subsystem: core::fmt::Debug {
    const SUBSYSTEM_NAME: &'static str;
    type Interface: InterfaceErrorKind = NoErrors;
    type Cascaded: ICascadedInner = NoErrors;
}

NoErrors is an uninhabited type (a Rust empty enum with no variants). It satisfies all the traits required to use it as interface or cascaded type (InterfaceErrorKind, ICascadedInner, GetRootCause, etc.) with bodies that are all unreachable!() — the compiler guarantees they can never execute.

This serves as the neutral element of sorts: subsystems that produce no interface errors or no cascaded errors specify NoErrors as the corresponding associated type. The Subsystem trait provides these as defaults:

pub trait Subsystem: core::fmt::Debug {
    const SUBSYSTEM_NAME: &'static str;
    type Interface: InterfaceErrorKind = NoErrors;
    type Cascaded: ICascadedInner = NoErrors;
}

While the code looks cumbersome, it's not going to be an issue once we add a DSL to generate all the boilerplate for us.

When I started working on improving the error handling, there were lots of error defining enums scattered across subsystems. In order to gradually migrate every subsystem to the new error types, I first introduced a type SystemError which is a simplified version of SubsystemError. It does not allow distinguishing between defects and interface errors, and it does not allow error wrapping. However, because it is also not subsystem-specific, all such errors are compatible regardless of their origin.

pub enum SystemError {
    LeafDefect(InternalError),
    LeafRuntime(RuntimeError),
}

SystemError is convertible from any SubsystemError<S> where S has Interface = NoErrors and Cascaded = NoErrors.

impl<S> From<SubsystemError<S>> for SystemError
where
    S: Subsystem<Interface = NoErrors, Cascaded = NoErrors>,
{
    fn from(value: SubsystemError<S>) -> Self {
        match value {
            SubsystemError::LeafUsage(_) => unreachable!(),
            SubsystemError::LeafDefect(internal_error) => internal_error.into(),
            SubsystemError::LeafRuntime(runtime_error) => runtime_error.into(),
            SubsystemError::Cascaded(_) => unreachable!(),
        }
    }
}

As I worked through subsystems, I gradually encoded explicit interface errors, until all of them were using SubsystemError.

The last piece from our system engineering overview was the error context. We would like to add arbitrary context to any error so that the human (or AI agent) has better understanding of what went wrong.

A popular Rust library anyhow allows attaching context to errors this way:

let content = fs::read(path)
        .with_context(||
                      format!("Failed to read instrs from {}",
                              path.display()))?;

We are not designing a generic system, so we can use the information that we have three kinds of stakeholders requiring different level of details in the context:

• For the proving system, we can't afford rich context at all; all diagnostics should be kept to an absolute minimum to save gas on proving. Besides, it does not support neither std nor the standard Rust allocator.
• For the forward run system, which executes ZKSync OS and the transactions on EVM, we either want:
  • a medium-sized context for a regular operation,

Additionally, we will fix a context format as a mapping from string keys to values.

Users can attach context to errors this way:

// Attach context lazily (noop on RISC-V)
result.with_context(|| error_ctx! {
    "caller" => debug_format(call_request.caller),
    "target" => debug_format(target),

    #[detailed] "extra_state" => expensive_computation(),
})

The error_ctx! macro is the end-user DSL for building contexts. Invalid usages are caught with compile_error! inside the macro arms:

• Empty string keys ("" => value)
• Unknown attributes (#[some_random_attribute])
• Duplicate #[detailed] attributes
• Identifier followed by => (misuse of shorthand)
• Missing => in key-value pair

This allowed my fellow ZKSync OS developers to have more guidance in using the newly added error_ctx macro.

I use two compiler time parameters to control the verbosity of the context actually included in the compiled binaries:

1. The target platform: RISC-V vs native
2. Cargo feature detailed_errors.

Additionally, we can also turn on and off error origin tracking.

# zk_ee/Cargo.toml
default = ["error_origins", "detailed_errors"]
error_origins = []    # Track file path + line number for errors
detailed_errors = []  # Enables #[detailed] context entries

Let's go quickly through our implementation of context.

The trait IErrorContext is the abstraction layer. On non-RISC-V targets, where we can use dynamically allocated memory, the context is a heap-allocated vector map. On RISC-V targets, it becomes a unit struct.

pub trait IErrorContext {
    fn get(&self, name: &str) -> Option<&String>;
    fn push(self, name: &'static str, value: impl ToString,
            visibility: ValueVisibility) -> Self;
    fn push_lazy<F>(self, name: &'static str, f: F,
                    visibility: ValueVisibility) -> Self
    where F: FnOnce() -> String;

    fn to_vec(&self) -> Option<Vec<NamedContextElement>>;
    fn into_vec(self) -> Option<Vec<NamedContextElement>>;
}

// On non-RISC-V targets
pub struct ErrorContext { values: Vec<(&'static str, String)> }

// On RISC-V targets
pub struct ErrorContext;  // unit struct, zero bytes

The first context method push includes the entry if the ValueVisibility and feature flags allow it:

pub enum ValueVisibility {
    AnyForwardRun,   // Included in all non-proving runs
    DetailedOnly,    // Only with `detailed_errors` feature, never on RISC-V
}

fn push(mut self, name: &'static str, value: impl ToString,
        visibility: ValueVisibility) -> Self {
    match visibility {
        ValueVisibility::AnyForwardRun => perform_push_value(),
        ValueVisibility::DetailedOnly if cfg!(feature = "detailed_errors") =>
            perform_push_value(),
        _ => {}
    }
    self
}

The second method push_lazy conditionally calls the closure only when the entry should actually be stored:

fn push_lazy<F>(mut self, name: &'static str, f: F,
               visibility: ValueVisibility) -> Self
where F: FnOnce() -> String {
    let should_include = match visibility {
        ValueVisibility::AnyForwardRun => true,
        ValueVisibility::DetailedOnly => cfg!(feature = "detailed_errors"),
    };
    if should_include { let value = f(); self.values.push(...) }
    self
}

This ensures expensive computations (e.g., debug_format(complex_object)) are never evaluated unless both the target is native and detailed_errors is enabled. The tests explicitly verify this with atomic counters.

Critically, push_lazy does not call f. The closure is constructed by the caller but never invoked. The Rust compiler, seeing that the result is unused, eliminates the closure and its captured environment entirely via dead-code elimination. This guarantees zero cost even for complex expressions passed to the context.

This table summarizes how much diagnostics information is retained for different targets:

First, what did we achieve for the price of all this complexity?

1. Explicit error semantics: The four-way classification (LeafUsage, LeafDefect, LeafRuntime, Cascaded) forces callers to reason about error semantics at match sites. It is impossible to accidentally conflate a contract violation with a programmer defect or resource exhaustion.
2. Structural cascade tracing: The CascadedError chain provides an implicit stack trace through subsystem boundaries. Combined with GetRootCause, error diagnosis can identify both the origin and the propagation path.
3. Uniform macro interface: All error creation is macro-driven with automatic location capture, making accidental locationless errors impossible.
4. Incremental verbosity: The three-tier system (proving → production → debug) provides meaningful information at each level without retroactively changing semantics. Errors are always errors; only metadata richness changes.
5. Lazy evaluation of expensive context: The push_lazy / #[detailed] mechanism guarantees that expensive operations (e.g., full debug formatting of complex state) are never even evaluated unless the feature is active. No runtime guard is required.
6. Zero-cost abstraction in proving context: Compiling for RISC-V eliminates all error metadata machinery at compile time, stripping diagnostics to the bare minimum with optional file+line information of origin.
7. Type-safe subsystem boundaries: SubsystemError<S> is parameterised – the type system enforces that a subsystem can only generate interface errors it declared, and can only forward errors from child subsystems it explicitly listed in its cascade type. Mismatched errors are compiler errors.
8. DSL ergonomics: The define_subsystem! macro reduces boilerplate dramatically. Declaring a new subsystem with its full error vocabulary is a simple, short invocation. The error_ctx! DSL includes rich compile-time validation with actionable compile_error! messages for misuse, making adoption easier.

In my opinion, we met the core constraints of the main stakeholders: rich, structured, traceable errors during development and testing, with complete overhead elimination in the zero-knowledge proof environment. This is achieved through Rust's type system rather than runtime switches, making it both robust and maintainable.

Where would we go from there? System engineering suggests the following plan for an iteration:

• identify stakeholders and their requirements;
• create measurable validation criteria;
• after implementing, verify and validate your work.

If we take things a bit more seriously and pour more resources into the engineering process, we'd have to build something like this table:

Taking this road requires studying the declared errors themselves, not the framework for defining them, so it promptly exits the scope of our little story.

The measurability is complicated for any requirements related to ergonomics, robustness, design constraints. So it seems reasonable to limit with the expert opinion of the stakeholders themselves. In the short time I've spent working on this project before I left Matter Labs, I did receive a positive feedback from my colleagues. If I were to continue, I'd contact more stakeholders, for example, smart contract developers who use an environment with a running instance of ZKSync OS to develop their contracts. If all stakeholders are subjectively satisfied, then the job was done well.