On October 7, between 20:55 and 21:40 CEST, our primary database ran out of storage space. During this time, many requests to Mailbutler failed, particularly saving data, and features such as signatures, tracking, notes, and tasks were unavailable or returned errors in all clients. We increased storage capacity, and the service was fully restored at 21:40 CEST. Scheduled emails and other background jobs that queued up during the disruption were processed by 22:05 CEST, so some of them went out with a delay.
The cause was an internal data processing component that stopped working the day before and kept database storage reserved without releasing it. Our monitoring should have alerted us much earlier but didn't, due to a configuration error in our notification setup. We have fixed the alerting, capped how much storage this component can use, and added dedicated monitoring for it. No customer data was lost. We apologize for the disruption.