Streaming Event Architect

---
name: streaming-event-architect
description: Design the INTERNAL event/streaming data backbone between services and stores — topic/stream taxonomy, partitioning and key choice (ordering stated as "per key, unordered across keys"), delivery semantics stated honestly (at-least-once with idempotent consumers; "exactly-once" claims interrogated), consumer groups, retry/dead-letter policy with replay procedure, retention vs compaction, backpressure/lag handling, event-schema compatibility rules, and CDC ingestion from operational stores. This is the data PIPELINE inside the system; the external contract surface — public webhooks, partner event feeds, signed envelopes, tenant-scoped subscriptions — is api-event-architect and stays there. Use when designing internal event flow, choosing stream vs queue, fixing ordering/duplicate bugs, or designing CDC. Do NOT use for the audit trail (audit-log-architect), product-analytics event taxonomy (event-schema-architect), or deciding which workloads leave the operational store (operational-vs-analytical-splitter).
---

# Streaming Event Architect

**Terms:** CDC means change data capture, DLQ dead-letter queue, SLA
service-level agreement, DDL data definition language (statements that create
or alter tables), authz authorization, and ack acknowledgement. A poison
message is one that fails on every retry and must be set aside.

## Purpose

Event pipelines fail in ways request/response systems never do: messages
arrive twice, out of order, late, or land in a dead-letter queue nobody
drains, and every one of those is a DESIGN outcome, not bad luck. This
skill produces the internal streaming backbone design — topics, keys,
partitions, delivery semantics, consumer layout, DLQ and replay, retention,
schema compatibility, and CDC ingestion — with each guarantee stated
honestly (what is ordered, relative to what key; what deduplicates, and
where). It owns the pipe BETWEEN internal components. The moment an event
crosses to an external consumer — a partner webhook, a public feed — the
contract belongs to `api-event-architect`, and this skill designs only how
the internal stream feeds it.

## Use When

- Use when: designing or overhauling how services exchange events
  internally — new event backbone, stream vs queue choice, topic layout.
- Use when: duplicate processing, out-of-order effects, growing consumer
  lag, or an overflowing dead-letter queue point to delivery-semantics
  design debt.
- Use when: designing CDC (change data capture) from an operational store
  into streams — including when `operational-vs-analytical-splitter`
  prescribed CDC and the transport now needs designing.
- Use when: event payload schemas keep breaking internal consumers and
  compatibility rules (registry, additive-only policy) need defining.
- Do NOT use when: the events are a PUBLIC or partner-facing contract —
  webhook delivery, signed envelopes, subscription scoping, external
  versioning/deprecation policy are `api-event-architect`; this skill stops
  at the boundary where the internal stream hands events to that layer.
- Do NOT use when: the "events" are the compliance audit trail — actor/
  action/outcome records with tamper-evidence and retention are
  `audit-log-architect`, even if they physically ride a stream.
- Do NOT use when: deciding WHETHER analytics/reporting workloads should
  move off the operational store — that decision is
  `operational-vs-analytical-splitter`; this skill designs the CDC/event
  transport after the split is decided.
- Do NOT use when: the payload evolution question is about DATABASE
  schema (tables/columns with deployed readers) — that is
  `schema-evolution-planner`.
- Do NOT use when: the concern is the worker EXECUTION model that consumes
  the stream — per-job retry/backoff, visibility timeouts, DLQ ownership,
  and tenant fairness are `background-job-orchestration-architect`; or live
  push to CLIENT connections (subscribe-time authz, fan-out,
  reconnect/replay) — that is `realtime-subscription-architect`. This skill
  designs the transport both consume, not what runs on top of it.
- Do NOT use when: the events are product-analytics measurement events
  (naming taxonomy, tracking plan) — that is `event-schema-architect`; this
  skill designs the internal transport.

## Inputs to Inspect

1. The event inventory as it exists: what is published today, by whom,
   consumed by whom, at what volume and peak rate — from code, queue/topic
   configs, and any existing event docs.
2. Ordering and consistency requirements per flow: which effects must be
   sequenced (per entity? per tenant? globally?) and which are commutative.
3. Consumer failure behavior today: retry loops, DLQ existence and drain
   procedure (or absence), poison-message history.
4. The operational stores feeding CDC: write rates, transaction
   boundaries, and whether ordering across tables matters downstream.
5. Tenant model constraints (from `multi-tenant-data-architect` outputs if
   present): whether streams are tenant-mixed with keyed isolation or
   tenant-separated, and what tenant context a tenant-scoped event carries;
   global events record an explicit unknown/not-applicable tenant state.
6. Existing schema/contract conventions: any registry or payload versioning
   already in force, and `api-event-architect` artifacts for events that
   ultimately feed external surfaces.

## Workflow

When a human must choose a backbone path such as replayable stream versus
work queue or global ordering versus throughput, define unfamiliar terms and
each viable option and why it fits. Compare case-specific ordering, replay,
throughput, failure handling, money, setup time, and ongoing upkeep; state
unknown costs instead of guessing. Recommend one from the flow evidence and
explain why it fits the user's goals and what fact would change the
recommendation before one atomic decision question. Ask for a missing
deciding fact first when needed. Keep routine design choices within this
skill; a preference does not authorize broker provisioning or
implementation.

1. **Inventory flows and classify each.** For every producer→consumer
   flow: fire-and-forget notification, state-transfer (fact/event-carried
   state), or command. Volume, peak rate, and fan-out. Misclassified
   commands-as-events are a standing source of coupling — flag them.
2. **Choose the transport shape per flow.** Durable log/stream (replayable,
   multi-consumer, ordered per partition) vs work queue (competing
   consumers, ack/redeliver) — by fan-out, replay need, and ordering need.
   State the choice per flow, not one global answer.
3. **Design keys and partitions.** A partition key is necessary for a
   per-key ordering design, but producer order, routing, broker retry
   settings and consumer handling must preserve it. State the actual
   guarantee and non-guarantees; use sequence checks and idempotency when
   concurrent producers or retries can reorder records. Key by the entity
   whose effects must sequence (commonly
   entity-id; tenant-id alone creates hot partitions for large tenants —
   note skew). Estimate partition counts from peak rate and consumer
   parallelism, with headroom for rebalancing.
4. **State delivery semantics honestly.** Default: at-least-once delivery
   with IDEMPOTENT consumers (dedup key = event id or business key;
   dedup-store retention stated). Interrogate any "exactly-once"
   requirement: end-to-end exactly-once across arbitrary side effects is
   not purchasable — what exists is transactional produce/consume within
   one platform plus idempotent effects at the edges. Design the
   idempotency mechanism per consumer.
5. **Design failure paths.** Retry policy (attempts, backoff, what
   distinguishes retryable from poison), dead-letter destination per
   consumer group, DLQ ownership (who is paged, drain SLA), and the REPLAY
   procedure: how a drained/fixed message re-enters, how a consumer replays
   a time/offset range, and what makes replay safe (the same idempotency
   from step 4).
6. **Set retention and compaction.** Per topic: time/size retention for
   event history vs compaction for latest-state-per-key; replay window
   consumers can rely on; storage cost note. Compaction changes semantics
   (intermediate events vanish) — mark topics where that is unacceptable.
7. **Define schema compatibility rules.** Registry discipline (schemas
   versioned and checked at produce time where tooling allows), additive-
   only within a major version, required-field addition = new major with a
   parallel-publish window, consumer tolerance rules (ignore unknown
   fields). Deprecation of an event version follows a stated window, and
   external-feeding events defer to `api-event-architect`'s contract.
8. **Design CDC ingestion where prescribed.** Capture mechanism
   (log-based preferred over polling — state why per store), transaction
   boundary handling, initial-snapshot + streaming handoff, and schema
   drift behavior (upstream DDL vs the stream schema — coordinate with
   `schema-evolution-planner` when the upstream table evolves).
9. **Address backpressure and lag.** Lag as a first-class signal per
   consumer group (alert threshold and owner — wiring via
   `observability-operator`, a manual-only skill a person runs by name), producer
   behavior when downstream is slow (buffer/block/shed, stated), and scaling
   rules for consumers.
10. **Deliver the design** in the Output Format, with per-flow guarantees
    and every handoff named.

Decision tables (stream vs queue, key choice, retention vs compaction,
CDC mechanisms) and the per-flow guarantee card format:
[references/backbone-decision-sheet.md](references/backbone-decision-sheet.md).

## Output Format

When asking for this build choice, show each option's meaning, reason,
case-specific benefits and limits, money/setup/upkeep or unknowns, and
a justified recommendation before the one decision question.

```
STREAMING BACKBONE DESIGN — <system/domain>
Flows:          <N producer→consumer flows, classified>
Per-flow card:
  <flow>: transport=<stream|queue> key=<key> ordering=<per-key guarantee,
          producer/broker/consumer assumptions and non-guarantees>
  delivery=at-least-once + idempotent consumer (dedup=<mechanism>)
  retry=<policy> DLQ=<dest, owner, drain SLA> replay=<procedure>
  retention=<time/size|compacted> schema=<version, compat rule>
Partitioning:   <counts, skew notes (tenant hot-key risk)>
CDC:            <sources, mechanism, snapshot policy, drift handling> | n/a
Backpressure:   <lag thresholds/owners; producer behavior when slow>
Boundaries:     external contracts → api-event-architect; audit trail → audit-log-architect;
                split decision → operational-vs-analytical-splitter (consumed, not made here)
Explicit non-guarantees: <what this design does NOT promise (e.g., cross-key ordering)>
```

## Validation Checklist

- [ ] User-facing build choices explain terms, reasons, costs or unknowns,
      pros/cons, and a justified recommendation before one atomic question.
- [ ] Every flow has a stated ordering guarantee in "per <key>, unordered
      across <scope>" form — no unscoped "ordered" claims.
- [ ] No unqualified "exactly-once" anywhere; every consumer has a named
      idempotency mechanism with dedup-store retention.
- [ ] Every consumer group has retry policy, DLQ destination, DLQ owner,
      and a replay procedure — no dead-letter black holes.
- [ ] Retention/compaction stated per topic with the replay window
      consumers may rely on.
- [ ] Schema rules cover both directions: producer evolution policy and
      consumer tolerance policy.
- [ ] Tenant context: tenant-scoped events carry it; global events mark
      unknown/not-applicable explicitly; hot-partition risk from tenant
      keying is assessed.
- [ ] The external boundary is explicit: which events feed
      `api-event-architect` surfaces and where the handoff happens.
- [ ] Explicit non-guarantees section is present and honest.

## Gotchas

- "Ordered" without a scope is a lie waiting to page you: per-partition
  order also depends on producer order, retry settings, routing and consumer
  handling. Cross-key and cross-topic sequencing requires design
  (sequence numbers, sagas), not assumption.
- Exactly-once marketing: platform transactions cover produce/consume
  within that platform; the moment a consumer touches an external system,
  you are back to at-least-once + idempotency. Say so in the design.
- Tenant-id as partition key concentrates your largest tenant on one
  partition — the noisy-neighbor problem reappears INSIDE the pipe. Key by
  entity within tenant, or accept and monitor the skew.
- DLQs without owners are where events go to die silently; a DLQ is only a
  safety mechanism if drain has an owner and an SLA.
- Replay without idempotency is an incident generator — replaying a day of
  events double-applies every non-idempotent effect.
- Compaction deletes history by design: a consumer needing every
  transition cannot read a compacted topic. Classify topics before
  enabling it.
- CDC initial snapshots on large tables can hammer the source store —
  snapshot strategy is part of the design, and its execution belongs in a
  `data-migration-runbook-author` runbook when it is a one-time move.
- Schema registries enforce shape, not meaning: a field renamed via
  delete+add passes additive checks while breaking every consumer
  semantically. Compatibility rules need review, not just tooling.

## Stop Conditions

- The events under design are actually a partner/public contract (signing,
  subscription management, external versioning) → stop and route to
  `api-event-architect`; designing them here creates a second, conflicting
  contract surface.
- A hard "exactly-once end-to-end" requirement survives interrogation as a
  business mandate → halt and surface the honest options (transactional
  scope limits + idempotent edges) to a human; do not paper over it with
  an unqualified guarantee.
- Ordering requirements conflict with throughput requirements (global
  ordering demanded at a volume one partition cannot carry) → present the
  tradeoff explicitly and stop for a human decision.
- The tenant-isolation posture for streams is undefined (mixed streams vs
  per-tenant separation has compliance implications) → obtain the tenant
  model from `multi-tenant-data-architect` outputs or a human before
  fixing keys and topics.
- Asked to also implement broker/platform configuration changes on live
  infrastructure → decline execution; this skill designs, and live config
  changes follow the repo's approval path.

## Supporting Files

- [references/backbone-decision-sheet.md](references/backbone-decision-sheet.md)
  — stream-vs-queue decision table, key-choice and skew analysis, retention
  vs compaction table, CDC mechanism comparison, per-flow guarantee card.
- `evals/evals.json` — behavior cases including the ordering-scope edge and
  the exactly-once refusal.
- `evals/trigger-evals.json` — discrimination against `api-event-architect`
  (THE seam), `audit-log-architect`, `operational-vs-analytical-splitter`,
  and `schema-evolution-planner`.