Sales Demos Orchestrator

---
name: sales-demos-orchestrator
description: "Install Automation Orchestrator and the PostgreSQL it cannot run without, so an environment ends up with a working orchestrator UI rather than a catalog entry. Deploys CloudNativePG, builds the three databases Temporal actually needs, installs the AO operator, creates the instance, and proves the Route serves a page. Runs playbooks/install_ao.yml. TRIGGER when: the user asks to install, deploy or fix Automation Orchestrator or AO, asks why the orchestrator UI is missing or 503s, says ao-temporal-migration is crash-looping, or built an environment with install_ao=false and now wants it. SKIP: if the user wants the AAP self-service portal — that is sales-demos-portal — or wants to install OpenShift Virtualization, which is sales-demos-setup."
---

# sales-demos-orchestrator

Installs Automation Orchestrator into an RHDP environment, database and all.
Takes about **5 minutes**.

This skill contains **no logic**. All the work is in
[`playbooks/install_ao.yml`](../../../playbooks/install_ao.yml). See
`CLAUDE.md` → *Skills and playbooks*.

**`setup.yml` already runs this on every build**, so reach for this skill when
you need it on its own: an environment built with `-e install_ao=false`, or an
AO stack that needs reconciling without re-running CNV and the whole AAP
configuration.

## What it does

1. Installs **CloudNativePG** (`certified-operators`, `stable-v1`) into
   `cnpg-system` with an AllNamespaces OperatorGroup.
2. Creates a single-instance PostgreSQL `Cluster` (`ao-db`, 10Gi) in
   `automation-orchestrator`, whose `initdb` makes the **backend** database.
3. Creates the other **two** databases as CNPG `Database` resources.
4. Reshapes CNPG's generated credentials into the two secrets the AO CRD wants.
5. Installs the **Automation Orchestrator operator** (`redhat-operators`,
   `stable`) and creates the `AutomationOrchestrator` CR with a Route.
6. Waits for `Ready=True`, then **asks the Route for a page** and requires a
   `200`.

**Both operator Subscriptions use `installPlanApproval: Manual`** (#593). The
playbook approves each operator's first InstallPlan, so a fresh install
completes. It never approves an upgrade, so a new AO build cannot land
unattended, which has already broken SSO once elsewhere. Manual does not pin a
version; a new environment still gets the channel head. To take an upgrade
deliberately:

```bash
oc get installplan -n automation-orchestrator-operator-system   # or cnpg-system
oc patch installplan <name> -n automation-orchestrator-operator-system \
  --type merge -p '{"spec":{"approved":true}}'
```

**After approving an AO operator upgrade, re-run
[`/sales-demos-orchestrator-config`](https://github.com/ericcames/sales.demos/blob/main/.claude/skills/sales-demos-orchestrator-config/SKILL.md).**
Its `APP_*` settings live in the `ao-admin-settings` ConfigMap, which the
operator does not own, so they should survive the upgrade — but that is
untested until one is taken, and re-running proves it (#608, #621).

## Three databases, not two — the thing that will waste your afternoon

The CRD requires exactly two secretRefs, `backendDatabase` and
`temporalDatabase`. Build two and `ao-temporal-migration` crash-loops forever:

```
pq: database "temporal_visibility" does not exist
```

Temporal keeps its visibility store in a **separate** database with a **fixed**
name — literally `temporal_visibility`, not a suffix of whatever you called the
temporal database. Nothing in the CRD, the sample CR, or the operator
description mentions it. The playbook creates all three. **Do not remove the
third because the CRD only asks for two.**

## Why it gets its own database

`aap-postgres-15` is owned by the `AnsibleAutomationPlatform` CR with
`blockOwnerDeletion` — the AAP operator reconciles it, so databases added by
hand live inside something another operator recreates at will. Temporal is
write-heavy, and putting that load on the database the whole demo platform runs
on trades a working AAP for a working AO.

## Logging in

**Username `admin`, password = this environment's AAP admin password.** The
playbook seeds it from `aap_password` so AO and AAP are one credential rather
than two (#143). It reads `aap_password`, never `env_secrets[<env>]` directly: on
a laptop `connection.yml` resolves that variable from the vault, and from AAP the
"Sales Demos - Env Secrets" credential type injects it, because a job template
has no vaulted file to read. Reaching for `env_secrets` worked from a laptop and
failed from AAP — the comment above the assert in `playbooks/install_ao.yml` has
the detail.

That is **seed-time only**. The CRD is explicit: the secret "is used only during
initial database seeding to create the admin user. Once the admin user exists,
this secret is ignored — password changes must be made via the application API
or CLI." So it fixes every environment built from now on, and does nothing to
one that already exists.

On an environment built **before** #143, the operator generated a random
password instead. Retrieve it with:

```bash
oc get secret ao-initial-admin-password -n automation-orchestrator \
  -o jsonpath='{.data.password}' | base64 -d
```

To change it afterwards, it is an API call, not a redeploy:

```bash
TOK=$(curl -sk -X POST "https://$AO_HOST/api/v1/auth/login" \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<current>"}' | python3 -c 'import json,sys;print(json.load(sys.stdin)["access_token"])')
AOUID=$(curl -sk "https://$AO_HOST/api/v1/users/me" -H "Authorization: Bearer $TOK" \
  | python3 -c 'import json,sys;print(json.load(sys.stdin)["id"])')
curl -sk -X PATCH "https://$AO_HOST/api/v1/users/$AOUID" \
  -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
  -d '{"password":"<new>"}'
```

`UID` is a readonly builtin in bash — name the variable something else, as
above, or the substitution silently uses your Unix uid and the PATCH 422s.

**Never paste the password into an issue, PR, or commit.** This repo is public.

## Preflight Check

```bash
./utilities/preflight.sh "${ENV:-sandbox}" --k8s
```

If any check fails, stop and tell the user exactly which one and the fix shown
beside it. Do not attempt the run with a failing prerequisite.

## Confirm the operators are actually offered here

Both come from catalogs that may differ between environments. Check before
running rather than discovering it three minutes in:

```bash
ENV=${ENV:-sandbox}
test -f ".kube/${ENV}.kubeconfig" || bash utilities/make-kubeconfig.sh "$ENV"
export KUBECONFIG=".kube/${ENV}.kubeconfig"

for pkg in cloudnative-pg automation-orchestrator-operator; do
  src=$(oc get packagemanifest "$pkg" -n openshift-marketplace \
        -o jsonpath='{.status.catalogSource}' 2>/dev/null)
  [ -n "$src" ] && echo "✅ $pkg offered by $src" \
                || echo "❌ $pkg is NOT in this cluster's catalog"
done
```

## Collect inputs

| Variable | Default | Meaning |
|---|---|---|
| `ENV` (inventory limit) | `sandbox` | Which environment to target — `sandbox` or `demo` |

The playbook's other inputs (namespaces, channels, database names, the 10Gi
volume) are vars with working defaults. Override them only for a reason.

## Run

```bash
mkdir -p ~/ansible-logs
export ANSIBLE_LOG_PATH=~/ansible-logs/install-ao-sandbox-$(date +%F-%H%M).log

./utilities/run-ansible.sh playbooks/install_ao.yml -i inventory --limit sandbox -e target_env=sandbox \
  --vault-id sales.demos@~/secrets/.vault_pass_sales_demos
```

**Always set `ANSIBLE_LOG_PATH`** — the log is the only evidence left if it
fails. Logs live outside the repo, in `~/ansible-logs/`. Tell the user the path.

**Never pipe the run through `tee`.** In a pipeline the exit status comes from
`tee`, not `ansible-playbook`, so a failed run reports success.

Tell the user this takes about 5 minutes and stream the output.

## Verify it in the EE before merging a change

See `/sales-demos-verify-ee` for why and how. The one command:

```bash
utilities/run-in-ee.sh playbooks/install_ao.yml \
  -i inventory --limit sandbox -e target_env=sandbox \
  --vault-id sales.demos@~/secrets/.vault_pass_sales_demos
```

## Verify on the cluster

**A green playbook run is not proof**, though this playbook already asks the
Route for a `200` before it reports success. Confirm independently with the
`openshift-sandbox` (or `openshift-demo`) MCP tools:

1. `pods_list_in_namespace` for `automation-orchestrator` — expect the database
   plus backend, UI, worker, background-worker, temporal and redis all
   `Running`, and the migration Jobs `Completed`.
2. `resources_get` the `AutomationOrchestrator` named `ao` and check
   `Ready=True` / `Degraded=False`.
3. `resources_list` Routes in `automation-orchestrator` and open the host.

Then open the URL. It should serve the Automation Orchestrator UI.

## When it finishes

Report the playbook summary **and** the verification above, then give the user
the URL.

## If it fails

| Symptom | Cause | Fix |
|---|---|---|
| `ao-temporal-migration` pods in `Error`, everything else waiting | The `temporal_visibility` database is missing | Re-run — the playbook creates all three. If it persists, check the `Database` resources report `status.applied: true` |
| `401` / `Unauthorized` on the first task | RHDP bearer token expired | Refresh `openshift_api_token` in the vault, re-run `make-kubeconfig.sh` |
| CSV wait times out | Wrong catalog for this cluster | CNPG is in `certified-operators`, AO in `redhat-operators`. Run the catalog check above |
| `no matches for kind "Cluster" in version postgresql.cnpg.io/v1` | CNPG CRDs not established yet, or you are looking at ODF's | ODF ships a vendored CNPG under `postgresql.cnpg.noobaa.io` — a different API group that will not serve these. Wait for the real CRDs |
| Route returns 503 | Router has no backend yet | Normal shortly after deploy; the playbook retries. If it persists, check the `ao-ui` pods |
| `Attempting to decrypt but no vault secrets found` | `--vault-id` missing from the command | Add `--vault-id sales.demos@~/secrets/.vault_pass_sales_demos` |

## Removing it

The whole stack, database included, is two deletes plus the operator:

```bash
oc delete automationorchestrator ao -n automation-orchestrator
oc delete namespace automation-orchestrator      # takes the database with it
```

The 10Gi PVC goes with the namespace. `teardown.yml` deliberately leaves all of
this alone — AO is setup-time infrastructure like CNV, not per-demo state.

Never paste a live cluster hostname or token into a commit message, issue, or
PR. This repo is public — see `CLAUDE.md`.