We are pleased to confirm that ilifu services will return online at 12:00 SAST on Wednesday, 16 September 2026, including access to the production CephFS filesystem.
If you are using the temporary NFS service, please finish your running jobs and log out before 12:00 on Wednesday. Your data on this temporary storage will remain available for a few weeks so that you can transfer it to your usual working area on CephFS. We will communicate the migration deadline and arrangements separately.
Following extensive recovery and verification work, a small number of metadata inconsistencies remain, affecting a very small fraction of the approximately 1.1 billion files on ilifu. Further recovery attempts for these remaining cases would require keeping ilifu completely offline and unavailable for all users for another 3-4 weeks, with a low probability of success. Having assessed the likely benefit against the impact of extending the outage for all users, we have decided to restore service and address the few outstanding issues separately. For those small numbers of users impacted, we hope you can understand why we have come to this decision. We will be reaching out to those we have identified as being impacted separately, however, if you encounter any file system anomalies, please report them to support@ilifu.ac.za.
This outage resulted from a highly unusual and unfortunately timed confluence of factors. It has reinforced the need for stronger operational controls, responsible use of shared resources by users and project leads, and additional redundancy measures. We will be reviewing our file-transfer policy and the approval process for research groups, particularly for transfers involving large numbers of files. Storage allocations will be enforced more strictly: once access is restored, please review your usage against your project allocations and delete what you don't need - you'll be improving ilifu's overall resilience in the process.
We again apologise for the extended outage and recognise the disruption and stress it has caused researchers, particularly students and postdocs approaching thesis or paper submission deadlines. We appreciate your patience and the sustained efforts of the recovery team. Working together, we can strengthen the resilience of ilifu while supporting the diverse needs of our research community. We also look forward to sharing news of new projects and expanded support in the coming months.
