op independently, through three separate caches that don't
coordinate, with the failover token unset, retry logic that retries the very error it should back
off from, and a warm-up job that fires into an active outage. So we blow through the daily
read limit and everything 429s until the 24-hour window rolls.
Every time a fleet process needs a secret (a Cloudflare token to deploy, a Brevo key to send email,
ChiroHD login for stats), it shells out to the 1Password CLI (op). Under load op
returns HTTP 429 "Too many requests". When that happens and nothing has cached the secret
yet, the work fails outright — that is what took down the Athena front-desk chat and repeatedly
blocked deploys today.
Source: /tmp/fleet-cred-get.log
pattern counts (independently re-verified by Codex). "Hard outage" = COLD-MISS-FATAL:
secret needed, cache empty, op capped.
| Plan tier | Read / hour (per token) | Write / hour | Combined / 24h (whole account) |
|---|---|---|---|
| Individual / Families | 1,000 | 100 | 1,000 |
| Teams | 1,000 | 100 | 5,000 |
| Business | 10,000 | 1,000 | 50,000 |
op whoami returns url: https://my.1password.com
(the default host, suggestive of Individual/Families), but whoami does not expose the plan
tier, and Codex's independent check hit a transient network error. Action: confirm the tier in
the 1Password account UI. It is the single fact that decides upgrade-vs-migrate. Working
hypothesis: Families, ~1,000/day.
Six layered causes. RC1 is the ceiling; RC2–RC6 are why we hit it so easily and recover so badly. Verdicts in brackets reflect Codex's independent file-level review.
If Families, ~1,000 reads/day shared across the entire fleet is nothing. Outage mechanism confirmed; exact tier still needs UI confirmation.
Every caller hits op independently; nothing coalesces concurrent requests. Five agents needing the CF token in the same second = five API calls for one secret. vault-load bulk-fetches many secrets in one burst.
Sage initially said two. Codex corrected: ~/.secrets/.op-cache is a fresh-op-binary copy (a separate dyld-wedge fix), not secrets. The real secret stores are three: macOS Keychain fleetcred (fleet-cred-get), ~/.secrets/.vault-cache (vault-get), and ~/.secrets/cache files (cred-cache-lib). Good news: cred-cache-lib already shares the fleetcred Keychain — so the real fragmentation is vault-get's separate cache. Plus a real bug: the Keychain key is item::field with no vault, so it can collide across the Automation and Codex/Claude vaults.
OP_SERVICE_ACCOUNT_TOKEN_B is unset everywhere (env, launchctl, keychain). 189 "would have failed over but couldn't" log lines. The headline resilience feature has never once engaged. (Caveat: a 2nd token only raises the hourly limit, not the daily cap.)
warm-all refreshes 24–25 creds in sequence with no health-probe gate, so it fires all of them even when op is already capped — 119 such skipped attempts logged.
op_retry retries on "nonzero exit + empty stdout" — and a 429 (printed to stderr, which is discarded) looks exactly like that, so it gets retried (3 total attempts). It does not inspect the 429 text. Narrower than first stated: fleet-cred-get itself does not use op_retry (it detects rate-limit text and backs off), but vault-load and others do.
op-reset-detector.sh probes for recovery, then fires refresh-fleet-critical-caches + multiple fleet-cred-get refresh loops the moment op comes back. A thundering herd at recovery that can immediately re-trip the limit.vault-get double-calls on a miss — tries the Automation vault, then retries with no vault, doubling the cost of every cold miss.op callers than Sage counted — the MCP wrappers (mcp-late-dev, mcp-brevo-ttc), refresh-brevo-ttc, vault-load, bus/send-chat.sh, skill-quality-transcript-eval.py, create_bot.py, fleet-activity-report.py, and ve-gsc-poll-and-capture.sh all call op directly, bypassing the cache.The decisive question. The current broken system attempts far more than 1,000 reads/day (uncached fan-out × retry tripling × stampedes × warm-in-storm). After the architecture fix, the fleet's steady-state demand collapses:
| Source of op reads (post-fix) | Reads / day |
|---|---|
| One health-gated daily warm of the critical-cred list (25 creds) | ~25 |
| Refreshes as 24h TTLs expire (mostly covered by the warm) | ~10–25 |
| Cold misses for non-warmed secrets, coalesced by single-flight | ~10–40 |
| Agent / MCP / cron restarts reading from a warm shared cache (0 op calls if cache is fresh) | ~0–30 |
| Realistic steady-state total | ~50–150 / day |
op_retry wrapper (×3) is a ×6 retry multiplier on every call.
A ~247-read cold-start storm becomes ~1,482 requests/storm; if it repeats every 5 min
during op-binary / auth churn, ~17,784/hr — which blows past even Business's
10,000/hr-per-token limit. So Business buys headroom; it does not make this
never-again on its own. The actual guarantee = a global circuit-breaker + single-flight + killing
the ×6 multiplier. Do that and the spike never forms; the tier is just insurance on top.
tier = headroom · the de-amplify + circuit-breaker = the guarantee — see §8
Three plans. The architecture fixes (unify caches, single-flight, governor, 429-safe retry, kill the stampede, one health-gated warm) are mandatory in all three — they are the actual root-cause fix. The plans differ only in where the secrets live and how much headroom we buy.
Do the full architecture fix, then upgrade the plan to Business for 50,000 reads/day of headroom.
vault-get delegates to fleet-cred-get; fix the item::field cache key to include the vault.op_retry classifies 429 and does zero retries; warm-all + cred-cache-lib abort the batch on first 429.OP_SERVICE_ACCOUNT_TOKEN_B for hourly fallover.Never-again? Strong yes — ~100/day need vs 50,000/day cap = 300–1000× headroom. Effort: architecture work (needed anyway) + ~1–2h to upgrade billing. Net new cost: 1Password Business seats (see §6).
Same architecture discipline, but move secrets to Infisical and run the Infisical Agent. Note: persistent local Agent caching is currently Kubernetes-only — the documented Agent caching is for dynamic-secret leases, and the Agent also retries failed requests 3× by default (same failure class as ours unless explicitly tuned). So the Agent is not a free fix.
vault-get + fleet-cred-get + MCP wrappers + refresh scripts, run a parallel-validation window.Never-again? Yes if self-hosted, but only because we'd still do the same architecture fix — and we'd add a new self-hosted server + the Agent as fresh single-points-of-failure. Effort: ~10–20h migration. Risk: lateral — new vendor, new failure surface.
Do the architecture fix, buy no upgrade, governor sized to the current (small) cap.
Never-again? Risky — on Families' 1,000/day, a bad spike (mass restart + cache wipe) can still graze the ceiling. Relies on caching being perfect with thin headroom. Good as the free first step, not the final answer.
| Plan | Price | Daily read cap | Fit for the fleet |
|---|---|---|---|
| Families (likely current) | $4.49/mo billed annually (~$54/yr, up to 5 people) | 1,000/day | Works only with perfect caching; thin headroom |
| Teams Starter Pack | $19.95/mo flat (10 members, billed annually) | 5,000/day | Comfortable — 33–100× headroom |
| Business recommended | $7.99/user/mo billed annually | 50,000/day (10k/hr) | Bulletproof — 300–1000× headroom |
Business cost depends on seat count. A handful of seats ≈ $8–$40/mo. Well within the under-$50/mo fleet budget. No code changes required for the upgrade itself.
| Plan | Price | Limits | Notes |
|---|---|---|---|
| Self-hosted (OSS) | $0 software (we host it) | Up to 5 machine identities, 3 projects/envs (free-tier feature gating); no vendor read cap when self-hosted | Cheapest on paper; real cost is our hosting + maintenance + a new SPOF |
| Cloud Free | $0 | 5 identities, 3 projects/envs, 10 integrations; 120 secret-ops/min (200 reads/min) | Per-minute cap is tighter than 1P's hourly window — a burst trips it faster, not slower |
| Cloud Pro | $18/mo per identity | 12 projects/envs, rotation, SSO, 90-day audit; 300 secret-ops/min (350 reads/min) | Per-identity pricing adds up fast; per-minute cap still tighter than 1P hourly on a spike |
| Enterprise | Custom quote | Dynamic secrets, gateways, KMIP, 99.99% SLA | Overkill for now |
.op-cache is a fresh-op-binary copy, not a secret
store; the real secret caches are three (Keychain fleetcred, .vault-cache,
~/.secrets/cache), and cred-cache-lib already shares the fleetcred
Keychain. Plan tier is UNVERIFIED — whoami can't prove it. RC6 is narrower than stated.op-reset-detector), vault-get's
double-call on miss, the item::field cache-key vault collision, and several uncached
direct-op callers (MCP wrappers, vault-load, send-chat.sh).
Codex built the worst-case spike from real config. The retry amplifier is the killer:
dead token-B (×2 attempts) × op_retry wrapper (×3) = ×6
on every call. A cold-start storm of ~223 live reads then becomes ~1,338 requests in one hour;
if it repeats every 5 minutes during op-binary / auth churn, ~16,000/hr — which
blows past even Business's 10,000/hr-per-token limit. So buying Business does not, by itself,
make this never-again. De-amplifying (kill the ×6) + a circuit breaker is what makes it never-again.
warm-all continuing after a 429 is codified in a unit test (fleet-cred-get.test.ts:41) — the fix must update the test too.op callers still live outside the proposed unified cache: bus.ts Slack + chat fallbacks (lines 1591, 1744), staff-meeting-transcribe.py:364, and others. The fix is not done until these are wrapped.op-reset-detector writes its sentinel after the refresh work already ran — no lock, so the stampede fires before anything guards it.op circuit breaker + single-flight — on the first 429, write a shared
"capped-until" flag, serve stale/cache-only everywhere, and allow exactly one governed refresh path.
"Anything else is decorating the blast crater."
op circuit breaker + single-flight. On the first 429, write a shared "capped-until" flag, serve stale/cache-only everywhere, and allow exactly one governed refresh path. This single change kills the stampede, the ×6 retry multiplication, and the warm-in-storm fan-out at once — with no new infrastructure. Then complete the rest of the architecture fix (unify caches, vault-aware cache key, 429-safe retry incl. the unit test, kill the recovery stampede, one health-gated warm, wrap the remaining direct-op callers). Mostly code we own; low-risk and reversible.