Some systems are experiencing issues

About This Site

This site will show any outages being experienced by the ilifu system.

Documentation for using ilifu is available here: https://docs.ilifu.ac.za/

Please log any issues you may experience using our support email address support@ilifu.ac.za

Stickied Incidents

5th June 2026

Access to ilifu cloud infrastructure

Dear colleagues

As hosts of the ilifu cloud infrastructure, the University of Cape Town (UCT), in coordination with the Inter-University Institute for Data Intensive Astronomy (IDIA), are issuing this advisory because SSH keys used to access ilifu may have been compromised. If you use SSH keys for access, please rotate them immediately and replace the old public keys on all systems and services where they are trusted.

SSH keys are commonly used across servers, code repositories, cloud platforms and automation systems. If a private key is exposed, an unauthorised person may be able to authenticate as you on any system that trusts the matching public key.

Please note that SSH private keys are not limited to the computer where they were created. If a private key is compromised, it may be used from another system wherever the corresponding public key is still trusted.

Required action

If you use SSH keys, please complete the following steps:

  1. Generate a new SSH key pair.
  2. Add the new public key to every system or service where you use SSH authentication.
  3. Test that the new key works as expected.
  4. Remove the old public key from all systems and services where it was previously trusted.

Systems and services that may be affected include Linux or Unix servers, bastion or jump hosts, code repositories such as GitHub, GitLab or Bitbucket, cloud platforms, automation or deployment systems, backup or monitoring systems, and any system where your public key appears in an authorised_keys file.

Best practice guidance

Please follow these SSH key hygiene practices:

  • Use separate SSH keys for different purposes, such as work, personal use, production access and code repositories.
  • Protect private keys with a strong passphrase.
  • Never share private keys with anyone.
  • Never send private keys by email, chat, ticketing systems or file-sharing platforms.
  • Do not store private keys in scripts, shared folders, source code repositories or documentation.
  • Remove old or unused public keys from systems and services.
  • Avoid SSH agent forwarding unless specifically required and approved.
  • Review your ~/.ssh/config, shell history, Git remotes, scripts and automation files for references to old keys.
  • Ensure private key files have restrictive permissions and are readable only by your user account.

What to do with old keys

After your new key is working, remove the old public key from every system or service where it was configured. Deleting the old private key from your workstation is not enough; the old public key must also be removed everywhere it was trusted.

If you are unsure where your SSH key is used or need assistance rotating it, please contact the relevant support team as soon as possible.

Should you have any questions relating to this email, please direct your query to idia-security@uct.ac.za.

Regards

University of Cape Town, in coordination with the Inter-University Institute for Data Intensive Astronomy

Past Incidents

5th June 2026

Access to ilifu cloud infrastructure

Dear colleagues

As hosts of the ilifu cloud infrastructure, the University of Cape Town (UCT), in coordination with the Inter-University Institute for Data Intensive Astronomy (IDIA), are issuing this advisory because SSH keys used to access ilifu may have been compromised. If you use SSH keys for access, please rotate them immediately and replace the old public keys on all systems and services where they are trusted.

SSH keys are commonly used across servers, code repositories, cloud platforms and automation systems. If a private key is exposed, an unauthorised person may be able to authenticate as you on any system that trusts the matching public key.

Please note that SSH private keys are not limited to the computer where they were created. If a private key is compromised, it may be used from another system wherever the corresponding public key is still trusted.

Required action

If you use SSH keys, please complete the following steps:

  1. Generate a new SSH key pair.
  2. Add the new public key to every system or service where you use SSH authentication.
  3. Test that the new key works as expected.
  4. Remove the old public key from all systems and services where it was previously trusted.

Systems and services that may be affected include Linux or Unix servers, bastion or jump hosts, code repositories such as GitHub, GitLab or Bitbucket, cloud platforms, automation or deployment systems, backup or monitoring systems, and any system where your public key appears in an authorised_keys file.

Best practice guidance

Please follow these SSH key hygiene practices:

  • Use separate SSH keys for different purposes, such as work, personal use, production access and code repositories.
  • Protect private keys with a strong passphrase.
  • Never share private keys with anyone.
  • Never send private keys by email, chat, ticketing systems or file-sharing platforms.
  • Do not store private keys in scripts, shared folders, source code repositories or documentation.
  • Remove old or unused public keys from systems and services.
  • Avoid SSH agent forwarding unless specifically required and approved.
  • Review your ~/.ssh/config, shell history, Git remotes, scripts and automation files for references to old keys.
  • Ensure private key files have restrictive permissions and are readable only by your user account.

What to do with old keys

After your new key is working, remove the old public key from every system or service where it was configured. Deleting the old private key from your workstation is not enough; the old public key must also be removed everywhere it was trusted.

If you are unsure where your SSH key is used or need assistance rotating it, please contact the relevant support team as soon as possible.

Should you have any questions relating to this email, please direct your query to idia-security@uct.ac.za.

Regards

University of Cape Town, in coordination with the Inter-University Institute for Data Intensive Astronomy

28th May 2026

Slurm Login Node Access to slurm-login and tranfer restricted due to compromised security

We are currently experiencing compromised security on the ilifu slurm-login and transfer node. Access to these nodes has been restricted to contain the unauthorised access. The technical team is investigating the compromised security. We have been made aware that similar issues are impacting several other HPC and Cloud sites in South Africa and are part of a broader national problem.

  • Since the security incident last week, we have been working with UCT CSIRT to investigate the extent of the compromised systems and to understand if a data breach has occurred. Similar incidents have impacted several other HPC and Cloud sites. The investigation is still ongoing, and unfortunately we cannot resume operations until it is complete.

    We believe that the attack vector was compromised SSH keys. SSH access to ilifu has been restricted since Thursday, 28 May. We have removed all existing SSH keys from ilifu user accounts. We will send out information about account access and adding new keys when we are ready to resume operations on ilifu. In parallel to the investigation, we are working on implementing further enhanced security measures.

    We strongly recommend that all users cycle their SSH keys at other remote sites as soon as possible.

  • The technical team is continuing to investigate the extent of the compromised security issue that occurred on the slurm-login and transfer node. In parallel, we are working on a process to overcome any possible compromised accounts to prevent further unauthorised access, and implementing further enhanced security measures, such as MFA. The development and deployment of these processes are expected to require a number of days to complete. We don't expect the ilifu services to be available before mid-week next week.

  • We've made the decision to restrict access to additional ilifu services while the technical team investigates the issue. Access to Jupyter and the Visual Studio Code services has been removed. The CARTA and Globus services remain operational.

    • 27th May 2026

    Internet Access to ilifu Issue with external access to slurm-login and transfer

    We're currently experiencing an issue with external connectivity on slurm-login and transfer node. The technical team is currently investigating the issue.

  • Access to slurm-login and transfer node has been restored. As before, running interactive sessions (via srun/sinteractive) or persistent terminal sessions (tmux, screen) will have been impacted. Please review your sessions and resubmit jobs where necessary.

  • We're again experiencing an issue with external access to slurm-login and transfer nodes. The issue has been escalated to the technical team. We are working to resolve the issue and restore access to these services. Apologies for the inconvenience caused.

  • Access to the slurm-login and transfer node has been restored. New slurm-login and transfer nodes were deployed, and in the case of the slurm-login node deployment, this would have impacted users running interactive sessions (via srun/sinteractive) or persistent terminal sessions (tmux, screen). Please review your sessions and resubmit jobs where necessary.

    • 7th May 2026

    Openstack Compute ilifu data centre offline due to power loss

    There was a power outage at the data centre building and the generator servicing the data centre did not start, so the data centre is currently offline. Technical support is currently on the way to the data centre to investigate the issue.

  • The ilifu facility is online and operational again. All ilifu services have been restored and are accessible again. Unfortunately, all running Slurm jobs were lost due to the power outage, please review job submissions and resubmit where necessary.

  • Power to the ilifu data centre has been restored. ilifu technical specialists are working to power on data centre servers and restore services.

    • 11th March 2026

    Slurm General operations The Slurm control daemon service is currently down

    The Slurm control daemon service is currently down. The systems team is investigating the issue.

  • The Slurm controller issue has been resolved and is now active. Users should be able to submit jobs and query Slurm accountting information again.