CephFS metadata repair requires suspension of ilifu services Friday 28th August 2026 19:04:00

Since the CephFS and MDS issues earlier this week, we have been experiencing issues with CephFS metadata records, a critical component of the filesystem.

In order to repair the CephFS metadata, CephFS must be taken offline. This means that access to the ilifu cluster will be prevented and all queued and running jobs will be stopped. As this is critical, we will be taking the ilifu cluster down in the next 30 mins. We will restore operations as soon as possible.

We are investigating a repeat failure of our cephfs filesystem

Posted 1 month ago by admin

We have brought ilifu CephFS online again and resumed ilifu operations. ilifu services are now accessible again.

Posted 1 month ago by jeremy

The technical team is currently running an analysis on ilifu CephFS in order to determine the next steps for the CephFS metadata recovery. This process may take several hours, and ilifu services continue to be offline during this process.

Posted 1 month ago by jeremy

Ilifu services continue to be offline while the technical team works to resolve the critical CephFS metadata issue. We will provide further updates during the day.

Posted 1 month ago by jeremy