Systematic Debugging

# Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

- Test failures

- Bugs in production

- Unexpected behavior

- Performance problems

- Build failures

- Integration issues

**Use this ESPECIALLY when:**

- Under time pressure (emergencies make guessing tempting)

- "Just one quick fix" seems obvious

- You've already tried multiple fixes

- Previous fix didn't work

- You don't fully understand the issue

**Don't skip when:**

- Issue seems simple (simple bugs have root causes too)

- You're in a hurry (rushing guarantees rework)

- Manager wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. **Read Error Messages Carefully**

- Don't skip past errors or warnings

- They often contain the exact solution

- Read stack traces completely

- Note line numbers, file paths, error codes

2. **Reproduce Consistently**

- Can you trigger it reliably?

- What are the exact steps?

- Does it happen every time?

- If not reproducible → gather more data, don't guess

3. **Check Recent Changes**

- What changed that could cause this?

- Git diff, recent commits

- New dependencies, config changes

- Environmental differences

4. **Gather Evidence in Multi-Component Systems**

**WHEN system has multiple components (CI → build → signing, API → service → database):**

**BEFORE proposing fixes, add diagnostic instrumentation:**

```

For EACH component boundary:

- Log what data enters component

- Log what data exits component

- Verify environment/config propagation

- Check state at each layer

Run once to gather evidence showing WHERE it breaks

THEN analyze evidence to identify failing component

THEN investigate that specific component

```

**Example (multi-layer system):**

```bash

# Layer 1: Workflow

echo "=== Secrets available in workflow: ==="

echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script

echo "=== Env vars in build script: ==="

env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script

echo "=== Keychain state: ==="

security list-keychains

security find-identity -v

# Layer 4: Actual signing

codesign --sign "$IDENTITY" --verbose=4 "$APP"

```

**This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)

5. **Trace Data Flow**

**WHEN error is deep in call stack:**

See `root-cause-tracing.md` in this directory for the complete backward tracing technique.

**Quick version:**

- Where does bad value originate?

- What called this with bad value?

- Keep tracing up until you find the source

- Fix at source, not at symptom

Phase 2: Pattern Analysis

**Find the pattern before fixing:**

1. **Find Working Examples**

- Locate similar working code in same codebase

- What works that's similar to what's broken?

2. **Compare Against References**

- If implementing pattern, read reference implementation COMPLETELY

- Don't skim - read every line

- Understand the pattern fully before applying

3. **Identify Differences**

- What's different between working and broken?

- List every difference, however small

- Don't assume "that can't matter"

4. **Understand Dependencies**

- What other components does this need?

- What settings, config, environment?

- What assumptions does it make?

Phase 3: Hypothesis and Testing

**Scientific method:**

1. **Fo