Workflow Orchestration Patterns

# Workflow Orchestration Patterns

Master workflow orchestration architecture with Temporal, covering fundamental design decisions, resilience patterns, and best practices for building reliable distributed systems.

Use this skill when

- Working on workflow orchestration patterns tasks or workflows

- Needing guidance, best practices, or checklists for workflow orchestration patterns

Do not use this skill when

- The task is unrelated to workflow orchestration patterns

- You need a different domain or tool outside this scope

Instructions

- Clarify goals, constraints, and required inputs.

- Apply relevant best practices and validate outcomes.

- Provide actionable steps and verification.

- If detailed examples are required, open `resources/implementation-playbook.md`.

When to Use Workflow Orchestration

Ideal Use Cases (Source: docs.temporal.io)

- **Multi-step processes** spanning machines/services/databases

- **Distributed transactions** requiring all-or-nothing semantics

- **Long-running workflows** (hours to years) with automatic state persistence

- **Failure recovery** that must resume from last successful step

- **Business processes**: bookings, orders, campaigns, approvals

- **Entity lifecycle management**: inventory tracking, account management, cart workflows

- **Infrastructure automation**: CI/CD pipelines, provisioning, deployments

- **Human-in-the-loop** systems requiring timeouts and escalations

When NOT to Use

- Simple CRUD operations (use direct API calls)

- Pure data processing pipelines (use Airflow, batch processing)

- Stateless request/response (use standard APIs)

- Real-time streaming (use Kafka, event processors)

Critical Design Decision: Workflows vs Activities

**The Fundamental Rule** (Source: temporal.io/blog/workflow-engine-principles):

- **Workflows** = Orchestration logic and decision-making

- **Activities** = External interactions (APIs, databases, network calls)

Workflows (Orchestration)

**Characteristics:**

- Contain business logic and coordination

- **MUST be deterministic** (same inputs → same outputs)

- **Cannot** perform direct external calls

- State automatically preserved across failures

- Can run for years despite infrastructure failures

**Example workflow tasks:**

- Decide which steps to execute

- Handle compensation logic

- Manage timeouts and retries

- Coordinate child workflows

Activities (External Interactions)

**Characteristics:**

- Handle all external system interactions

- Can be non-deterministic (API calls, DB writes)

- Include built-in timeouts and retry logic

- **Must be idempotent** (calling N times = calling once)

- Short-lived (seconds to minutes typically)

**Example activity tasks:**

- Call payment gateway API

- Write to database

- Send emails or notifications

- Query external services

Design Decision Framework

Does it touch external systems? → Activity
Is it orchestration/decision logic? → Workflow

Core Workflow Patterns

1. Saga Pattern with Compensation

**Purpose**: Implement distributed transactions with rollback capability

**Pattern** (Source: temporal.io/blog/compensating-actions-part-of-a-complete-breakfast-with-sagas):

For each step:
  1. Register compensation BEFORE executing
  2. Execute the step (via activity)
  3. On failure, run all compensations in reverse order (LIFO)

**Example: Payment Workflow**

1. Reserve inventory (compensation: release inventory)

2. Charge payment (compensation: refund payment)

3. Fulfill order (compensation: cancel fulfillment)

**Critical Requirements:**

- Compensations must be idempotent

- Register compensation BEFORE executing step

- Run compensations in reverse order

- Handle partial failures gracefully

2. Entity Workflows (Actor Model)

**Purpose**: Long-lived workflow representing single entity instance

**Pattern** (Source: docs.temporal.io/evaluate/use-cases-design-patterns):

- One workflow execution = one entity (cart, account, inventory ite