Tracing the Origin of a Real Life MySQL Hack
How to find the origin of attack if your MySQL DB was hacked
Incident
I was testing the deployment of one of my projects by hosting it on a AWS virtual machine. I had hosted both my database and the backend on this. I was yet testing the deployment without any serious data and therefore I hadn’t implemented any security measures for the VM or the DB. One day all of a sudden I see my app complaining that my tables are missing. So I logged into the database to see what was the issue and I find this message on a table called WARNING in my DB.
Poor Guy! He must have thought this was a production DB and I’ll come running to him asking to take back the data 😅 (All the data are public information and does not contain any sensitive materials, so there was nothing to worry. I also had a backup). Anyway, this made me rethink about the security of the dev deployment. At the testing phase we don’t think much about the security and focus on getting the things working ASAP even by bypassing security sometimes. But lesson learned! I would NEVER do that again. I would be implementing at least the basic security even if my project is in the basic stages.
Vulnerabilites
First I’ll note down the security measures I overlooked and then I’ll walk you through how I traced the origin of the hack
- Leaving all the ports in the VM open — I had some issues with networking, so I decided to open all the ports one day so that I don’t have to worry about opening and closing ports
- Leaving the root MySQL with its default credentials
- Using a weak password for my MySQL user
- Allowing external access to the MySQL server — I have a multi server deployment which requires external access to the MySQL server
- Not enabling general logs for MySQL — This is not compulsory if you do not want auditing capabilities
Ok, looking at this list its no wonder that you see me as stupid. But believe me almost all the students do not consider about security at all in the initial phase of the project and that is exactly why I am telling this story so that at least one of you would not make this mistake again.
Fixes
These are just basic security measures that you can use to improve the security of your host. Yes, it slows down you a bit, but it doesn’t really take that long and I highly recommend that you do it. I think the major concern is not that it takes time but not taking security threats seriously. So here I am asking you to take it seriously even if it is a dev server.
- Close all the unnecessary ports in the host — Planning attacks on exposed ports are one of the first things a hacker does. So make sure you expose only the essential ports to the outside.
- Remove unwanted services and update the system regularly — Security vulnerabilities in services are used frequently to gain access to the system
- Always use STRONG passwords and use different password for different account
- Change the MySQL root user’s password and change the username ‘root’ of the admin user(both MySQL and the host) — Often bruteforcing passwords is done to gain access to systems with default or weak usernames and passwords
- Allow accessing MySQL server only from the localhost. If it is a multi server deployment, make sure you whitelist the IPs of the clients and restrict all other traffic
- Enable general logs for MySQL — Only error logs(which log errors like failed login attempts) are logged by default. This makes it harder to track a hacker due to lack of logs. Enabling general logs could cause a performance degradation though, so it can be used if you suspect a foul play
https://www.freecodecamp.org/news/cjn-is-your-mysql-secured-7793e5444cf5/ has some good tips on the steps that you can take to secure the MySQL DB
How to identify the origin of the Hack
Disclaimer: I am not a security expert. The following sections explains what I have done to identify the origin of the hack and the conclusions that I have made according to my knowledge on penetration testing and the observations that I came across on the hacked server.
I googled the message that the hacker posted for me and seems that I am not the only one.
There are lot of tutorials on how to hack a servers, but I couldn’t find a decent tutorial on how to identify the origin of the hack, if you MySQL database has been hacked. That is another reason for this blog post and I hope this will help another person facing the same issue.
To recover your lost databases and avoid leaking it: visit http://***.onion and enter your unique token *** and pay the required amount of Bitcoin to get it back. Databases that we have: ***db. Your databases are downloaded and backed up on our servers. If we dont receive your payment in the next 9 Days, we will sell your database to the highest bidder or use them otherwise. To access this site you have use the tor browser https://www.torproject.org/projects/torbrowser.html | http://***.onion | ***- My first observation was that the hacker has been able to access only a single DB
(***db). This DB was specially created for this project and I created a single user who has all the privileges for this DB. Since other databases are untouched, its highly possible that the hacker got access only to this user. - Unless the hacker used a vulnerable service on an exposed port, it is pretty hard to hack the SSH login. I have a strong password for SSH, so it is not possible to be bruteforced as well. Anyway to confirm this, I checked the history of MySQL commands executed in the VM.
Everything looks normal and shows the commands that I’ve used recently. So I think I can rule out possibility of a hacked VM. This was probably a MySQL hack done remotely without logging into the VM and that is why you should restrict remote access from unknown IPs.
3. I checked whether the general logs are available and they are available only for the last day. So that didn’t give me any successful results. However, if you have enabled the general logs you can find the access logs in the table general_log in the database called mysql or in the file defined with the variable general_log_file in the mysql.cnffile. Add the following in the config file and restart the MySQL server if you want to enable general logs.
[mysqld]
log_output = FILE
general_log = 1
general_log_file=/path/to/file/general.log4. Since I am out of luck on the general logs, I thought of checking the error logs. Unfortunately error logs contain only the failed login attempts, so I can’t pinpoint the exact moment of the hack using this. But I can check whether there have been any failed attempts to access the DB. The error log can be usually found in /var/logs/mysql/error.log
The latest error logs do not show anything unusual except someone from zg-0708a-27.stretchoid.com trying to access the DB. Lets dig deeper into older logs to see more. Note that the files have been archived using gunzip and you can use the command gunzip <archive_name>.gz to get the original file.
and Voila! Here we have the culprit. Without the general logs we can’t exactly say whether this guy was successful but owing to the weak password I had for the user I am pretty sure the bruteforce succeeded.
A little note if you are a hacker though: Try the default password for root user before trying to bruteforce somebody’s DB for days. It might save you a lot of time 😉 ( Speaking from the experience because this guy has spent days trying to bruteforce his way in while a simple root:root attempt would have succeeded)
One more weird thing was the guy from strechoid.com (When I say guy it would mean a bot as well). Visiting their URL lands me on a pretty weird web page.
These guys are saying that they are researching on online services and allowing us to opt-out. But I am definitely not going to give my name and email to somebody who crawls other people’s host without a proper explanation and without at least a convincing website 😆. Anyway seems this one is also a pretty common problem and there is no way to say whether these are actual researchers or hackers posing as researchers.
Back to the guy who attempted the brute force, I searched the location of the IPs that have involved with the scene. But there is no much information you can extract from the location because he could have easily used a VPN or multiple IP addresses to mask his exact location.
I don’t have enough knowledge on tracing back IPs to its original location, so I thought this would be a good stopping point for spending time on this. It was a pretty basic hack once you got to know the IP of the VM and I don’t have a need to pay the guy to get back the data, so there won’t be a use in digging this up further. Rather than panicking and creating a new deployment, I wanted to convert this to a learning opportunity to discover how to trace the point of the origin of the hack.
Since lot of guys have faced this issue and haven’t been able to figure out what happened I posted this article. I hope it will help them and contribute to improving the sense of cyber security in the readers. I highly recommend implementing the security measures I have discussed above next time you do a even a testing deployment. A production deployment need lot more security measures than that, so better leave it to an expert. 😃
