2023-01-12 Loss of Network Access Postmortem

Key people: Brendan (silver).

What happened

On 2023-01-12 we received an email from ITD stating that 193.1.99.123 was registered as an attack source by Heanet and external access was removed by ITD to contain it.
Later the same day it had been upgraded to our entire subnet, the ITD security team saw our servers as a security risk.

What was the root cause

Unlike other incidents we knew from the outset what the root cause was.
One of the Wordpress instances on www.skynet.ie decided its true calling was to become a spambot, this is what caught the attention of Heanet.

Restoring network access

In order to restore network access ITD had two requirements:

  1. Servers are patched on both an OS & application level.
  2. Maintained in such a manner to prevent unauthorized access or misuse

Rebuilding services

Due to the age of the software on virtually all machines in place upgrades were neither feasible nor maintainable.

As a result user data and config files were backed up from the servers that would be reused in the future.
Other servers were archived in case of a need to get more data in the future.

Using the newest server with adequate hard drive bays Proxmox was installed.
Various containers were then created to serve the roles needed to run the cluster.
The OS chosen for these containers was NixOS, a config based operating system, which allows us to easily update with consistency.

This process took until the end of April.
We were then able to request that ITD open specific ports for servers.
Over the summer more services became active.

Outcomes

As a result of all this: