Summary

Occasionally, a small number of requests to the StackGuardian API timed out. The cause was an intermittent software fault that, from time to time, left one of our API servers unable to finish some requests until it was automatically restarted. Our other servers kept handling requests normally. The permanent fix went live in production on October 8, 2026 at 17:55 UTC.

The incident on October 8, 2026 is one example. Between 07:36 and 08:45 UTC that day, 112 requests that create or change resources timed out. The problem cleared on its own when the affected server was restarted. This report uses that incident to explain what happened.

Nothing was wrong with your Terraform code or your configuration.

What you may have seen

• Occasional timeouts, mostly on create, update and delete requests. The affected requests ended with a 504 Gateway Timeout. Requests handled by our other servers succeeded as normal, so most requests worked while a few failed. In the October 8 example, requests that only read data were not affected.
• A "400 Bad Request" error in Terraform. The StackGuardian Terraform provider reported the timeout as a 400 Bad Request, for example "Error reading Role Assignment". Running the plan again worked.
• Changes that were made although an error was returned. The requests that timed out had already done their work before they got stuck. If a request timed out for you, the resource may still have been created or updated.

Cause

A fault in a third-party library we use could leave a request stuck after it had saved its changes but before it replied. Our own code triggered the fault through the way it handled some connections. Once one request was stuck, later create, update and delete requests on the same server waited behind it and received no reply until the server was restarted. The server kept responding to other requests, so it was not detected and replaced sooner.

The 400 Bad Request in Terraform was a second, separate fault. When the provider retried a timed-out request, it sent an incomplete request, which the API rejected as invalid.

What we fixed

The two changes that stop the server fault went live in production on October 8, 2026 at 17:55 UTC:

• We upgraded the affected library to a version without the fault. We could reproduce the fault on the old version and cannot on the new one.
• We changed how our API handles these connections, which removes the trigger.

The Terraform provider now sends the full request when it retries, so a brief server error is retried properly and no longer shows up as a 400 Bad Request. This fix is done in code and ships in the next provider release after v1.12.4.

We have not seen the fault recur since the fix went live.

How we are preventing it from happening again

The fixes above remove the fault we found. Two further measures guard against the same kind of problem from any other cause.

• A time limit on every API request. A request that runs too long will be stopped, so one stuck request can no longer hold up others, whatever made it stick. This is built and tested and is in review; it is not live yet.
• Automated tests for both fixes. Tests now check the connection handling in our API and that the Terraform provider's retry sends the full request.

We apologize for the disruption this caused.