A user was abusing the service yesterday. They were the reason redis ran out of memory. Their account was partially deactivated after sending them an email. They then returned late in the day causing the abusive activity to start again. At this point their account was removed but some incorrect attempts were made to reduce pressure on the system.
One of the incorrect attempts was to purge redis. API tokens are stored in redis without graceful recreations. Tokens have been backfilled. This is certainly a point of fragility in the design of this system. Reworking the backend system is on the agenda for 2023 and this fragility will be looked at.