There have been some performance issues related Ceph fs, specifically due to the metadata server (Ceph MDS). This problem shouldn't affect all users, and is often path specific e.g. the time taken for a ls -la command at a specific folder might take much longer than usual. We are busy investigating the issue.
25th April 2025
Jupyter SpawnerJupyterhub spawner down
We have received multiple reports that the Jupyterhub spawner is not working. We are busy investigating the issue.
The main JupyterHub server was temporarily unavailable due to exhausted network connection resources. The server has been rebooted, and all JupyterHub services are now restored and operational.
17th April 2025
ilifu sevices down
We are investigating a problem with all ilifu services that are currently down
We have identified the fault that caused services to be offline and have resolved this.
All systems should now be online
17th February 2025
CephFS mounted FilesystemsCephFS experiencing slow client IO due to full disks
We're currently experience slow client IO on the CephFS mounted filesystem on ilifu, as part of the process of Ceph balancing data across disks. Users maybe experience this as longer than normal job runtimes as write speeds are currently being throttled. The technical team is working to resolve this issue and restore client IO performance.
We have identified a faulty hard drive in the Ceph storage pool. This disk has now been replaced. The necessary data re-balancing following this hardware change has finished. We are now monitoring Ceph's performance until Monday (21 April) to ensure that the main causes of issues have been resolved.
The CephFS filesystem is currently experiencing degraded write performance. The technical team is working to resolve the issue.
Ceph write performance has returned to normal. The technical team will continue to monitor the situation.
