Now we have got all the servers back online, I would like to take this time to talk about our backend breach and answer questions you guys have had about it over the past few days.
On the 22nd of January, our first server to be taken offline was Colony. We first thought it was just a normal server crash, but when going into the server files found that everything had been wiped. We later had Discord messages from few people demanding money for the Colony server files, or they would also attack our other servers. We weren’t ever planning to give in to the demand, I’m also not sure who would think that someone would pay for a Minecraft world file 😀
After the server files where removed, we acted quickly to lock down all user accounts on the panel, change all root password and investigated a way they could have got into the backend of our servers. Finding nothing really amiss, a few staff members kept an eye on the server in case of a further breach and we downloaded backups from the servers.
Infinity, Mythic & Other Networks
On the 23rd, we woke to find all our servers had been taken offline and had other networks, like our partner the Arsenal Network, reporting to us that their servers had also been attacked and world files damaged.
Restoring The Network
Its taken around 2 weeks to get the all the servers back online and world files on all server have been reset. Due to an issue in the way our backups were being handled, any backup we had made had corrupted locations in the maps, meaning hours were wasted waiting for the world files to be reuploaded into the server, for them to have to be reset anyway. I also took this time to rebuild the backend of our servers, so that also extended the time our server where offline as well.
Cause of The Breach
We narrowed down two ways that the server could have been breached, one I’m not going to go into here, and the other is our panel. We have a web interface (panel) that allows for our staff team to easily access server files, send commands via the console, and keep an eye on chat. Much like you would if you rented a server from a Minecraft server host. Last Friday the makers of our panel announced they had found a huge security vulnerability and issued a quick patch to fix it. As our other networks who reported damage to their servers also used the same panel as us, this seems most likely how access to our server files was gained.
If you used the Pterodactyl panel like us, please read their blog post on the issue and update ASAP: https://blog.pterodactyl.io/pterodactyl-security-announcement-january-31st-2019/
Steps We’ve Taken
In case of anything like this happening again, we have massively improved the way we handled backups. We should now be able to restore a server in a matter of hours rather than days. As we’re just a small Minecraft network, we didn’t think that anyone would bother trying to get into our servers, so was not prepared when someone did. For this, I am sorry to all of our players!
Huge Thank You!
I would just like to say a huge thank you to all the players and other networks who supported us while we got the servers back online. We didn’t think we would have any players left after having our servers down for weeks, but some servers have now grown in player count, rather than fallen. So also a huge thank you for waiting for us.
Also, to all of the staff team who was there helping me get the servers back online. Whether you were helping me with backend work or building a spawn, it would have taken us way longer to get the server backonline with our your help, so thank you.
Note on Personal Data
No personal data was breached in this attack. All address, emails and names that you may have put into our online shop, is stored on its own server.