Debugging

---
name: debugging
description: >
  Finding the cause of a fault instead of a change that makes the symptom go away:
  reproducing before diagnosing, shrinking the reproduction until nothing is removable,
  stating a hypothesis that predicts an observation, changing one variable at a time,
  bisecting, and choosing which evidence to collect from a running production system before
  it is destroyed. Use when a fix is being guessed at, when a change "seems to work", when
  the same bug keeps coming back, when a fault cannot be reproduced, when a production
  incident needs a cause rather than a restart, when print statements are being added
  everywhere, or when several changes were made at once and it now works. Does not cover
  JVM performance triage (java-performance), GC (jvm-gc-tuning), live thread diagnosis
  (concurrency-diagnostics), heap dump mechanics (heap-dump-analysis), or deliberately
  injecting failures (distributed-systems-testing).
---

# Debugging

## Purpose

Turn a report into a cause. The alternative — changing things until the symptom disappears —
produces code nobody understands, a bug that returns under a slightly different input, and no
way to tell whether the change helped or moved the failure somewhere quieter.

## Workflow

Start with the requested outcome: diagnosis, mitigation or repair. Inspect the report, affected
call path, documented contract, existing tests and effective configuration before accepting the
report's explanation. Reuse adequate evidence; ask only about missing expectations or context
that could change the conclusion. Diagnosis can finish with a supported finding without editing
code. Behavior consistent with the accepted contract may call for clarification rather than a fix.

1. **Restate the fault as an observation.** "Customer 88123 saw a negative balance at 14:02"
   is an observation. "The refund logic is broken" is a hypothesis wearing a report's clothes,
   and adopting it early is how the wrong subsystem gets investigated for a day.
   State the expected behavior and its source too: an allowed overdraft is not evidence of a
   violated invariant. Separate requirements and established contracts from assumptions.
2. **Reproduce it.** Deterministically if possible, intermittently if not — but know which,
   because "I cannot reproduce it" and "it reproduces one time in ten" lead to different work.
   If it only happens in production, collect evidence within the mitigation budget
   (`references/production-evidence.md`).
3. **Shrink.** Remove inputs, steps, data and configuration until removing anything more makes
   the fault disappear, while preserving the original failure signature. Stop when the next
   reduction costs more than it helps; a minimal reproduction need not identify the full cause.
4. **State a hypothesis that predicts something you have not yet looked at.** "If the cause is
   the missing time zone, then the row written at 23:30 local will carry yesterday's date."
   Also name an observation that would refute it. Explaining existing evidence is useful;
   a discriminating prediction makes the next experiment useful.
5. **Run a controlled, discriminating experiment**, normally changing one variable. Preserve
   comparable state and a baseline control; record confirming and refuting outcomes. When an
   interaction is suspected, compare the relevant combinations deliberately instead of treating
   a successful batch of changes as an isolated cause (`references/method.md`).
6. **Check the explanation against the distribution** — frequency, timing, affected and unaffected
   users. Record remaining contradictions and uncertainty instead of forcing one cause to explain
   unrelated failures.
7. **When repair is in scope, verify it against the original fault.** Turn the reproduction
   into regression coverage when it earns lasting value (tdd); confirm it detects the original
   failure before accepting the corrected result. Reuse existing adequate tests. A race may need
   controlled scheduling or integration coverage. Record intermittent outcomes and relevant
   unaffected cases; do not make extra changes merely to finish a diagnosis-only task.

Inspect deployed versus source JDK/toolchain, dependencies, JVM flags, configuration and data
versions before version-sensitive diagnostics or changes. This workflow has no universal Java
baseline. Preserve the target environment; upgrading it is a separate decision. Return the fault,
evidence, tested/refuted hypotheses, fix or mitigation, validation and unresolved gaps. Missing
access or reproduction permits an evidence plan, not an invented root cause.

## Rules

- Prefer a causal explanation before a permanent fix. During an incident, a reversible mitigation
  may precede diagnosis; label it as mitigation, preserve the evidence the response budget allows,
  and do not present symptom disappearance as root-cause proof.
- Prefer one controlled variable per experiment. If a batch helps, it implicates the batch but
  does not isolate a member or interaction; reduce/revert the batch in a controlled environment.
- Read the whole stack trace, including cause and suppressed chains. The deepest application frame
  is a useful boundary candidate, not a verdict: framework callbacks, generated code, reflection,
  native frames, and library defects can move or hide the causal frame.
- Balance evidence preservation against customer impact and the incident commander's authority.
  Capture cheap, non-disruptive evidence first when the error budget permits; mitigate immediately
  when delay is unsafe, and record which volatile evidence the action destroyed
  (`references/production-evidence.md`).
- Symptom disappearance is evidence of recovery, not proof of cause or durable correction.
- Missing events refute a hypothesis only within verified collection coverage. Check enablement,
  filters, sampling/thresholds and workload opportunity before interpreting an empty result.
- Question the assumption that the fault is where the symptom is. Corrupted state is written
  in one place and observed in another, often much later. Trace the first contract violation;
  an incorrect write is one candidate, but valid data can also be misread or misinterpreted.
- Consider `git bisect` when verified good/bad revisions and a reliable classifier exist.
  It can narrow a monotone change in roughly log₂(n) classifications; build cost, skipped commits
  and intermittent or nonmonotone outcomes affect that benefit. See the reference's exit protocol.
- Delete the debugging output before the change ships, and if a log line was genuinely useful,
  promote it deliberately with a level and structure (structured-logging) rather than leaving
  a `System.out.println`.
- Timebox according to incident severity and experiment cost. Repeated refutations are a signal
  to revisit observations, harness assumptions and the system model or seek another perspective.

## References

- **The method in detail** — `references/method.md`. Shrinking a reproduction, differential
  diagnosis (what changed — code, data, config, dependency, traffic, time), bisection over
  commits and over data, reading exception chains, and the specific traps of intermittent and
  heisenbug faults. Read when the fault resists the workflow above.
- **Evidence from a running system** — `references/production-evidence.md`. What each source
  can and cannot answer — logs, metrics, traces, thread dumps, heap dumps, JFR, database state,
  deployment history — with volatility, cost and collection order during an incident. Read
  before touching a production system that is currently faulty.