Redis out of memory

Incident Report for Zeplo

Resolved

Things are operational. Still looking for any disrupted requests.

Posted Jan 24, 2023 - 16:07 UTC

Update

Keeping an eye on things

Posted Jan 24, 2023 - 14:16 UTC

Update

A workspace was generating an abusive amount of requests causing everything to backup. I have deactivated the account for the time being.

Posted Jan 24, 2023 - 14:12 UTC

Update

scaled up workers but that is causing too much load on the database

Posted Jan 24, 2023 - 13:58 UTC

Monitoring

Redis finished scaling

Posted Jan 24, 2023 - 13:10 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jan 24, 2023 - 12:59 UTC

Update

Scaling in progress

Posted Jan 24, 2023 - 12:59 UTC

Update

Scaling up instance

Posted Jan 24, 2023 - 12:52 UTC

Identified

Redis is out of memory causing requests to stay active

Posted Jan 24, 2023 - 12:49 UTC

This incident affected: API.