Lessons Learned from a Critical Data Breach: A Case Study

In today's digital age, ensuring robust cybersecurity and proper operational procedures is crucial for any business. This case study highlights how one small business overcame a significant data breach and transformed its technical infrastructure and policies for better security and efficiency.

The Situation

The client is a small business that builds web tools for other businesses.

Lacking in-house software engineers, the client hires engineers remotely through platforms like Upwork due to the lower cost of international engineers compared to those in the United States. They had active contracts with 2–4 remote engineers, each working on different aspects of the site.

The web application includes a MongoDB database and a backend application, both running on an EC2 instance in AWS. The client had only taken a few manual backups of the EBS volume attached to the EC2 instance and did not have an automated backup system in place. Additionally, there were several integrations with third-party services such as Twilio.

The Breach

In early May, the client experienced a severe data breach: the MongoDB database was deleted and replaced with a ransom note demanding ~$500 in Bitcoin. The client's last manual backup was from November, seven months ago, and lacked the latest user data. Consequently, the site was taken offline. The client suspected a recently dismissed engineer might be behind this.

The ransom note:

All your data is a backed up. You must pay 0.043 BTC to
1Kz6v4B5CawcnL8jrUvHsvzQv5Yq4fbsSv 48 hours for recover it.
After 48 hours expiration we will leaked and exposed all your data.
In case of refusal to pay, we will contact the General Data
Protection Regulation, GDPR and notify them that you store user
data in an open form and is not safe. Under the rules of the law,
you face a heavy fine or arrest and your base dump will be dropped
from our server!

Recovery

Initially, things looked bleak. The database was wiped, there were no recent backups, the site was down, and nobody had a clear solution. It seemed our best option was to restore the database from the backup made several months prior.

Fortunately, one of the engineers had made a backup of the database just a few days before the incident. Relieved, we immediately verified the backup and restored it to production.

Proper Procedure

After the site was restored, we began working with the client to implement proper access control policies and a solid technical infrastructure. Specifically, we recommended they:

Migrate to AWS DocumentDB from self-managed MongoDB

• Managed Service: Reduces the burden of database maintenance and management
• Automatic Backups: Provides automated backups, reducing the risk of data loss

Significantly trim down AWS access

• Implement IP-address restrictions on SSH and DB access
• Provide remote engineers with limited SSH and DB access for development and deployment
• Remove all remote engineers' AWS Admin Access

Set up a staging environment and restrict production access

• Controlled Production Access: Only trusted personnel can deploy new code
• Staging Environment: Allows for testing without affecting production

Remove personnel access before terminating their work contract

• Ensures former employees or contractors cannot access company systems

Set up GitHub branch protection

• Prevents unauthorized or accidental changes to critical branches

Revise the interview process for potential engineers

• Helps in selecting more qualified and trustworthy engineers

Improve how API keys are handled

• Properly managing API keys reduces unauthorized access to third-party services

Explore the budget for hiring more experienced engineers

• More experienced engineers lead to better code quality and enhanced security practices

Conclusion

This experience was a wake-up call for the client and underscored the importance of having robust security measures and operational procedures in place. By taking these steps, the client not only recovered from the breach but also fortified their infrastructure against future threats, ensuring a more secure and efficient operation moving forward.