p95 latency above 300 ms on api-gateway

Monitoring SEV-2 INC-2291 Opened 13:07 UTC by alert Mitigated in 28m

Rolled back to 9c2e71b at 13:35. p95 has held under 200 ms for 1h 24m. Resolve once it stays under the SLO for 2 hours, or reopen if it climbs.

api-gateway latency

  • p95
  • p50
  • Deploy and rollback
  • UTC
a3f9c1e rollback

Timeline

  1. Mara Okafor moved the incident to monitoring

  2. Dani Reyes rolled api-gateway back to 9c2e71b

    6 of 6 instances healthy in 1m 52s

  3. Dani Reyes The idempotency cache grows without a cap; GC pauses line up with the spikes.

    api-gateway · iad1-3 · stderr
    13:21:07.412 WARN gc pause 412ms heap=1.84GiB idem_keys=418204
13:21:09.006 ERROR POST /v1/transfers 504 upstream timeout after 3000ms
13:21:09.113 WARN gc pause 389ms heap=1.86GiB idem_keys=419977

  4. Mara Okafor took incident command and paged Dani Reyes

  5. Alert p95 latency above 300 ms fired on api-gateway

    PagerDuty · #payments-oncall

  6. Dani Reyes deployed a3f9c1e feat(gateway): idempotency keys on POST /v1/transfers

Details

Service
api-gateway
Environment
production · iad1 fra1
Commander
Mara Okafor
Cause
a3f9c1edeploy
Peak p95
371 ms at 13:14
Errors
129 × 504
Requests hit
3.1% of POST /v1/transfers
SLO budget used
6.4% of 30d

Follow-ups

1 of 3 done
  • Cap idempotency cache at 50k keys9c2e71bDani Reyes
    Done
  • Alert on heap above 1.5 GiBMara Okafor
    Oct 30
  • Write the postmortemMara Okafor
    Oct 31
↑ ↓ to move · Enter to run · Esc to close