Acknowledgment
First and foremost, thank you for choosing BingoIDC’s products. We would like to provide the following clarifications and summaries regarding the severe fault incident that occurred on January 24, 2023, at the Miami storage node, to ensure your right to be informed.
Incident Overview
On the evening of January 23, 2023, two Seagate hard drives within the server located at the Miami main storage node experienced mechanical failures. One hard drive became disconnected, while the other was in a data protection state. Due to an error by the maintenance personnel in the data center, the second hard drive was removed, resulting in RAID failure and automatic system shutdown. This issue was detected by the automated monitoring system at 4:00 AM on January 24 (with a detection time of 5 minutes). The operational staff from the Miami vendor intervened immediately at 4:00 AM. Simultaneously, our chief operations administrator sent a notification email to all users of Miami products at 6:00 AM.
By 11:00 AM, the operational team in Miami confirmed that the recoverable data of the server unit had been fully restored. Utilizing the ZFS system, we confirmed by 1:00 PM that all data was intact. In adherence to our commitment to customer responsibility, we conducted two verification checks—both manual and automated—to ensure the integrity of every business storage item. We promptly transitioned services to the nearby Dallas data center to restore original operations.
Compensation Plan
In light of this incident, we have discussed and proposed the following compensation plan: in response to approximately 12 hours of downtime, we will offer all users compensation equivalent to one month of effective usage time as an expression of our apology. Additionally, you may choose to increase your bandwidth by 100 Mbps, depending on the content of your support ticket.
Application Process
To apply for compensation, please reply to our support ticket titled “Miami Compensation Confirmation.” Our staff will complete your specified request within 48 hours.
We sincerely apologize for this incident. We will terminate our contract with the Miami operator and host the self-assembled Western Digital storage servers in another data center in Miami.
Thank you for your support and trust in BingoIDC.
Sincerely,
Alex White
Eschscholzia News Center
Edon_M
Chief Operations Administrator