Filesystem performance issues due to Ceph MDS bug Thursday 23rd July 2026 11:50:00

We are currently experiencing an issue with the CephFS MDS (metadata server) which is impacting file system performance. Users may experience this as stalled processes or hanging command line prompts. This is predominantly affecting the /idia mount. We are working to resolve this issue. In parallel, we are performing a Ceph minor version upgrade to address the issue.

The Ceph minor version update was completed on 2 August. CephFS has completed its recovery process. Users should no longer experience filesystem performance issues.

Posted 1 month ago by jeremy

The MDS issue affecting /users has been resolved, users should be able to access services again.

Posted 2 months ago by jeremy

We're currently experiencing issues with the CephFS MDS service affecting /users (users' $HOME directories) and access to ilifu services. The technical team is working to resolve the issue.

Posted 2 months ago by jeremy

While the primary MDS issue has been resolved, CephFS is still in a recovery state and the upgrade to the latest minor version is still ongoing. Users may still experience filesystem performance issues which may sometimes impact access to ilifu services. The technical team is monitoring the environment and working to bring CephFS into a stable state.

Posted 2 months ago by jeremy

We experienced an issue with the CephFS MDS service last night, which resulted in issues with the CephFS mounts, preventing users from logging on to ilifu services. The technical team is currently working to resolve the issue.

Posted 2 months ago by jeremy

We are unfortunately still experiencing CephFS MDS issues and are working to resolve this. Users may still experience filesystem performance issues.

Posted 2 months ago by jeremy

The MDS issues have been resolved, and access to services have been restored. Users should no longer experience the filesystem performance degradation and hanging processes.

Posted 2 months ago by jeremy

The MDS issues are also currently impacting access and performance of the /users, /cbio and /ilifu mounts.

Posted 2 months ago by jeremy