Slow and failing requests on the US node

This incident has been resolved.

API requests on api.us1.notabene.id and transaction views on app.us1.notabene.id that needed Network data were slow or timed out. The EU node was unaffected.

Timeline:

• 6 Oct 14:38 UTC: Customer reports received. Incident opened.
• 6 Oct 14:51 UTC: Root cause identified as a missing network route and security rule between the US production cluster and the Network service.
• 6 Oct 15:14 UTC: Network rule applied. Requests recovered immediately. No backlog processing was required.
• 6 Oct 15:15 UTC: Recovery confirmed.