Lab: Workflow Execution — Build a Simple Mesh Workflow

What you'll build

Vertical stack context: DOSFIUniversity (research) → DOSFI Academy (this course) → MeshNativeExchange (publishing & monetizing). Students progress from Learning → Building → Publishing → Monetizing. In this lab, you will learn to orchestrate a distributed workflow across multiple Distributed Intelligence Units (DIU). You will design a three-stage pipeline (Ingestion, Inference, Output) using a formal schema, implement node-to-node communication, and simulate fault recovery when a node drops out during execution.

Skills you'll practice

• Design a structured JSON schema for a multi-stage distributed workflow
• Map workflow stages to specific DIU capabilities using node labels
• Visualize the execution trace through state transitions
• Implement state persistence to handle mid-workflow node failures

Before you start

• Understanding of JSON/YAML structures
• Basic knowledge of distributed messaging patterns
• Familiarity with the DIU (Distributed Intelligence Unit) lifecycle

Lab steps4 steps

1. 1
   Defining the Workflow Schema

   In this lesson, we define the DAG (Directed Acyclic Graph) for our workflow. We use a JSON manifest to declare stages. Every task requires an input dependency and a capability selector. Example schema: { 'workflow_id': 'proc-001', 'stages': [ {'id': 'ingest', 'task': 'load_data', 'requires': []}, {'id': 'inference', 'task': 'model_run', 'requires': ['ingest']}, {'id': 'render', 'task': 'format_output', 'requires': ['inference']} ] }. Each DIU compares its own capabilities against these requirements. We avoid static addressing in favor of dynamic capability matching to ensure high availability.

2. 2
   Mapping Stages to the Mesh

   Nodes in the mesh broadcast capability heartbeats. A DIU with high compute resources reports { 'cap': 'inference' }. We define a scheduling policy that matches workflow stages to the most idle node with the required capability. Diagram: [Ingest Node] --(stream)--> [Compute Node] --(results)--> [Render Node]. We look at how the orchestration engine assigns a 'lease' to a node for each task. If a lease expires without a heartbeat, the DIU controller reassigns the work.

3. 3
   Tracing Execution and State

   Tracing is vital in a distributed environment to avoid the 'black box' problem. We track state transitions at every boundary. Table: | Time | Stage | Input | Output | NodeID | Status |. We use distributed tracing headers (TraceID, SpanID) in the headers of our internal mesh RPC calls. This allows us to query the status of a partially completed workflow and determine exactly which node holds the current state buffer.

4. 4
   Fault Tolerance and Recovery

   Distributed systems must expect failure. In this lesson, we simulate a node drop-out at the 'inference' stage. When a node hangs, the watchdog timer triggers. The system must verify the state persistence of the last successful stage. We use an idempotent retry mechanism: the workflow engine rolls back to the last checkpoint (Ingestion Output) and broadcasts a re-assignment request. The mesh automatically reroutes the task to a healthy node without the user knowing the failure occurred.