Skip to main content

Workflow Chaining & Orchestration Patterns

Introduction​

As your automation library grows, individual workflows become components of a larger system. A data-fetching workflow you built last week becomes a data source for today's analysis workflow. Your backtest runner becomes a sub-process called by a screening workflow.

The Run Workflow worker is the glue that makes this possible. This guide walks through real orchestration patterns — when to use each mode, how to pass data, and how to build robust pipelines.

Mode Decision Guide​

Choosing between fire-and-forget and wait-for-result determines how your workflow flows.

Use Fire-and-Forget When​

  • The target workflow runs independently
  • You don't need its output to continue
  • You want parallel execution
  • You're triggering notifications or alerts

Example: After processing signals, trigger a Telegram notification workflow. You don't care about its result — you just want the message sent.

Use Wait-for-Result When​

  • The target workflow produces data you need
  • Steps must run sequentially (A → B → C)
  • You're transforming data through stages
  • You need to check if the target succeeded

Example: Fetch market data, then analyze it, then store results — each step depends on the previous one.

Pattern 1: Sequential Pipeline​

The most common pattern — a linear chain where each step depends on the previous.

[Trigger] → [Run Workflow "Fetch Data" (wait)] 
          → [Run Workflow "Process Signals" (wait)] 
          → [Run Workflow "Generate Report" (wait)]
          → [Telegram Notify]
graph LR
    T[Trigger] --> W1[Fetch Data]
    W1 --> W2[Process Signals]
    W2 --> W3[Generate Report]
    W3 --> N[Telegram Notify]

Why this works: Each Run Workflow node waits for its target to finish, extracts targetResults, and the next node can use them. The parent workflow acts as a conductor — it doesn't process data itself, it orchestrates.

Variable passing: Pass symbol, timeframe, and other parameters from the trigger down through each step:

{
  "symbol": "BTCUSD",
  "timeframe": "1h",
  "limit": 500
}

The "Fetch Data" workflow uses these to query the market. When it finishes, its output is captured. The "Process Signals" workflow can receive new variables or reference the fetched data.

Pattern 2: Fan-Out Parallel Execution​

Launch multiple workflows simultaneously — useful for multi-symbol analysis or running the same logic with different parameters.

[Trigger] → [Run Workflow "Analyze BTC" (fire-and-forget)]
         → [Run Workflow "Analyze ETH" (fire-and-forget)]
         → [Run Workflow "Analyze SOL" (fire-and-forget)]
graph LR
    T[Trigger] --> BTC[Analyze BTC]
    T --> ETH[Analyze ETH]
    T --> SOL[Analyze SOL]

Why this works: All three analysis workflows are independent. With fire-and-forget, the parent doesn't wait — they all start nearly simultaneously. Each runs in its own executor process.

Variable passing: Each Run Workflow node passes a different symbol:

{ "symbol": "BTCUSD" }
{ "symbol": "ETHUSD" }
{ "symbol": "SOLUSD" }

Pattern 3: Parameter Sweep Orchestration​

Combine the Parameter Loop worker with Run Workflow to test multiple configurations across a shared target workflow.

[Trigger] → [Parameter Loop (symbols)] → [Run Workflow "Backtest Strategy" (wait)]
         → [Sort/Filter Results]
graph LR
    T[Trigger] --> PL[Parameter Loop]
    PL --> RW[Run Workflow Backtest]
    RW --> SF[Sort & Filter Results]

Why this works: The Parameter Loop iterates over a list of values (symbols, timeframes, parameter combinations). For each iteration, it triggers the "Backtest Strategy" workflow with different variables. In wait-for-result mode, each iteration completes before the next starts — or you can run them all in fire-and-forget mode and collect results with a downstream aggregator.

Variable passing: The Parameter Loop sets the symbol, and Run Workflow passes it to the backtest:

{ "symbol": "{{ loop_value }}", "period": 14 }

Pattern 4: Conditional Orchestration​

Use a Condition worker to decide which workflow to trigger based on data.

[Trigger] → [Fetch Data] → [Analyze Volatility]
         → [Condition: volatility > threshold?]
              → Yes: [Run Workflow "Alert High Volatility"]
              → No: [Run Workflow "Normal Processing"]
graph LR
    T[Trigger] --> FD[Fetch Data]
    FD --> AV[Analyze Volatility]
    AV --> C{Condition<br/>volatility > threshold}
    C -->|Yes| HV[Run Workflow<br/>High Volatility Alert]
    C -->|No| NP[Run Workflow<br/>Normal Processing]

Why this works: The Condition worker evaluates a rule. Each branch triggers a different workflow — fire-and-forget is ideal here since you're routing rather than sequencing.

Pattern 5: Accumulator / Aggregator​

Run multiple data-gathering workflows in parallel, then combine their outputs.

[Trigger] → [Run Workflow "Fetch US Stocks" (wait)]
         → [Run Workflow "Fetch Crypto" (wait)]
         → [Merge Results] → [Run Workflow "Global Analysis" (wait)]
graph LR
    T[Trigger] --> US[Fetch US Stocks]
    T --> CR[Fetch Crypto]
    US --> MR[Merge Results]
    CR --> MR
    MR --> GA[Run Workflow<br/>Global Analysis]

Why this works: Both fetch workflows run in parallel (fire-and-forget). A downstream worker (e.g., Sumator or Python Exec) merges targetResults from both. Then the merged data is passed to "Global Analysis" via another Run Workflow node.

Variable Passing Best Practices​

What Gets Passed​

Variables passed to the target workflow become context vars — accessible as {{ variable_name }} in any subsequent node.

Variable Types​

TypeExampleNotes
Strings"BTCUSD"Single values work everywhere
Numbers14, 3.5Use for periods, thresholds
Arrays["BTC","ETH","SOL"]Useful with Parameter Loop downstream
Objects{"rsi": 14, "ma": 200}Group related parameters
Nested{"analysis": {"period": 14}}Access as {{ analysis.period }}

What NOT to Pass​

  • Large datasets — variables go into MongoDB run documents. Pass identifiers or short config, not megabytes of OHLC data.
  • Binary data — JSON only. Use the target workflow to fetch data independently.
  • Secrets — variables are stored in run history. Don't pass passwords or API keys.

Error Handling​

Timeout Handling​

In wait-for-result mode, if the target workflow doesn't finish within the timeout, the worker returns:

  • status: "timeout"
  • targetStatus: "running" or "pending"
  • message explains the situation
  • The triggered run is still running — it doesn't get cancelled

What to do: Connect the Run Workflow node's output to a Condition worker that checks for status == "timeout". If timed out, log it or trigger a notification. The target run can be monitored later via its run_id.

Failed Target Workflows​

If the target workflow fails, the worker returns:

  • targetStatus: "failed" or "error"
  • targetError: the error message
  • The parent workflow can continue — it doesn't crash

What to do: Branch on targetStatus. If failed, route to an error-handling workflow (e.g., log to User Lists, send an alert).

graph LR
    RW[Run Workflow<br/>wait-for-result] --> C{Condition<br/>targetStatus == finished}
    C -->|Yes| SUCCESS[Continue Normal Flow]
    C -->|No| ERR[Run Workflow<br/>Error Logging]

Performance Considerations​

Fire-and-Forget Overhead​

Each triggered run goes through:

  1. MongoDB write (~5ms)
  2. Redis queue push (~1ms)
  3. Executor pick-up (depends on queue depth)

Total overhead: typically under 100ms per trigger.

Wait-for-Result Overhead​

In addition to trigger overhead, the parent workflow polls every 1.5 seconds. Each poll is a lightweight MongoDB query (by indexed _id).

For a target workflow that takes 10 seconds, expect ~11.5 seconds total (10s execution + 1.5s polling overhead).

Parallelism Limits​

The executor pool processes runs as fast as it can. Fanning out 20 workflows won't overload the system — they queue and execute in order. If you need guaranteed parallel execution, use multiple Run Workflow nodes with fire-and-forget mode.

Real-World Example: Daily Trading Pipeline​

Here's a complete daily pipeline that a trader might build:

Schedule Trigger (daily at 8:00 UTC)
→ Run Workflow "Fetch All Portfolio Data" (wait)
  ├─ Fetches BTC, ETH, SPY, QQQ from Twelve Data
  └─ Returns cleaned OHLCV arrays
→ Run Workflow "Compute Technical Indicators" (wait)
  ├─ Adds RSI, MACD, Bollinger Bands, Ichimoku
  └─ Returns enhanced dataframe
→ Run Workflow "Generate Trading Signals" (wait)
  ├─ Applies strategy rules
  └─ Returns buy/sell/hold signals with confidence scores
→ Condition: any signals generated?
  ├─ Yes: Run Workflow "Place Orders" (fire-and-forget)
  └─ No: Run Workflow "Log No-Trade Day" (fire-and-forget)
→ Run Workflow "Send Daily Report" (fire-and-forget)
  ├─ Generates markdown summary
  └─ Sends via Telegram notification

Each of the five sub-workflows (Fetch Data, Compute Indicators, Generate Signals, Place Orders, Send Report) is independently testable, reusable, and maintainable. The parent workflow simply orchestrates them.

Summary​

PatternModeBest For
Sequential PipelineWait-for-ResultData transformation, multi-stage processing
Fan-OutFire-and-ForgetMulti-symbol analysis, parallel tasks
Parameter SweepBothStrategy optimization, symbol iteration
Conditional RoutingFire-and-ForgetDecision trees, alert routing
AccumulatorWait-for-ResultMulti-source data aggregation
Error HandlerWait-for-ResultRobust pipelines with fallback paths

The Run Workflow worker transforms your workflows from standalone scripts into composable, reusable components. Start with simple chains and build up to complex orchestration as your automation needs grow.