Detect attacks on Linux servers

Even if you harden your server to the most paranoid level, it’s always possible that an attacker may sneak past your defenses. Then what? That’s where IDS (intrusion detection systems) enter the picture. An IDS watches your logs and alerts you if any suspicious entries appear, making it much easier to detect attacks on Linux. Most IDS software watches the logs of a single host, or an entire network. In this article, we’ll create our own simple solution for watching logs from scratch. Still, we’ll link you to major solutions in case you want to play with established tools.

Once we’re set up, we’ll simulate an attack and see if our new detection system notices the attack and alerts us in time!

Searching for signs of hacking

First, we need to pick which files to scan, so we can detect attacks as they happen.

Let’s start with /var/log/auth.log. Any SSH login attempts will appear here. Use the command cat /var/log/auth.log to view all SSH login attempts on your server.

We can open a live stream of failed login attempts by adding the tail command into the mix, like so:

tail -f /var/log/auth.log \
  | grep -i 'failed'

For breach detection, though, we’re more concerned with successful logins. So let’s try this:

tail -f /var/log/auth.log \
  | grep -i 'accepted'

What does this command do? First, the tail -f part gives us a live feed of the auth log. We see all new logins as they occur. Then grep makes sure we only see successful login attempts, since those are the ones that matter most if we’re worried about detecting successful breaches.

But enough theory, let’s run this on the system and see for ourselves what happens.

root@hackingloops:~# tail -f /var/log/auth.log \
  | grep -i 'accepted'
Jan 12 02:07:01 hackingloops sshd[110308]: Accepted publickey for root from hackingloops.com port 61946 ssh2: RSA SHA256:mBmc1Xug/AqlynrmyqXZRhms7R5fBxcggdY9+TjT3Mw

A good start, but we can’t really expect to sit there staring at this log every hour of every day. How can we send an alert to our sysadmin when an unknown IP address connects?

We’ll do two things:

1. include a file full of known IPs that we want to ignore.
2. Set up an alert system to notify admins when an unknown IP logs in.

Excluding known IPs

Wouldn’t you get annoyed if you received an alert everytime a user logs on to the server? Even worse, what if a program were logging in via SSH all the time? You’d receive constant alerts. Aside from annoying you, this has real security concerns. If you’re constantly under a barrage of security alerts that don’t matter, you will likely miss the ones that actually do matter.

When users begin ignoring important info because they’re under a hailstorm of constant, pointless alerts, it’s called “alert fatigue”. Such fatigue is endemic to cybersecurity, where more and more monitoring means ever more alerts.

How should we avoid this?

The answer in our case is simple, let’s start by excluding known IPs. That way, admins won’t receive an alert when a known, expected IP address logs in via SSH. To make this happen, we just need to create a file called known_ips.txt and include each known IP on a new line, like so.

root@hackingloops:~# cat known_ips.txt
127.0.0.1
192.168.0.1
[etc - any other known IPs you want to ignore]

Now modify the tail command from above to use this new list.

root@hackingloops:~# tail -f /var/log/auth.log \
  | grep -i 'accepted' \
  | grep -vFf known_ips.txt

Great, we’re done with that feature! Now the final step.

Using encrypted alerts to help detect attacks on Linux

If we needed to, we could bother to write an entire alert system from scratch to use as an example. But luckily we won’t need to. We can reuse a tool we wrote in a prior HackingLoops essay: Forward Secrecy with Python.

First, we can download the code from Github and deploy the alert listener on the admin’s personal machine.

root@hackingloops:~# git clone https://github.com/darighost/forward-secrecy-poc-app.git
Cloning into 'forward-secrecy-poc-app'...
[...]
root@hackingloops:~# cd forward-secrecy-poc-app
➜  forward-secrecy-poc-app git:(main) flask run
 * Running on http://127.0.0.1:5000

That’s great, our server is now awaiting incoming alerts! Now all we need to do is just whip up the client so it sends our alerts…

root@hackingloops:~# tail -f /var/log/auth.log \
  | grep -i 'accepted' \
  | grep -vFf known_ips.txt \
  | python3 client.py

This will send the alerts to our admin machine. At least, it should. Let’s try it out and see what happens! Just login with SSH from an IP not listed in known_ips.txt and…

Great! We can see in the screenshot that the alert shows up on the admin’s laptop! Now we’ll be able to easily detect attacks on Linux servers running this script. Plus, as an extra bonus, we built it using our previous Python code, so it already has end-to-end encryption and forward secrecy built-in.

---

A quick, homemade fix like this works if you’re just learning and having fun on your own personal server. However, as a security engineer or blue teamer, this would not suffice. Instead, we would want a more robust solution with more features for finding and thwarting attacks as they occur.

For that reason, we should also explore some of the existing, commercial tools that attempt to solve this problem, and see which tools are best for which situations.

Learn more on how to detect attacks on Linux

Now that we’re done playing with our homecooked scripts, let’s see what the market has to offer. Plenty of enterprise tier products compete in the IDS market to help admins detect attacks on Linux, but when it comes to free and open source options, two systems reign supreme: Snort and Wazuh.

The different between the two is simple: Snort specializes in watching your network, and Wazuh is more focused on watching a specific machine. Of course, the line between the two is not always clear in real life, and this is only a generalization.

Although you can play with both of these products for free, neither is trivial to deploy. Still, attempting to do so is a great learning exercise, and you can find docs for installing them via these links:

• Install Snort: https://www.snort.org/#get-started
• Install Wazuh: https://documentation.wazuh.com/current/installation-guide/

Moreover, note that merely installing these tools is only the start. Once you’ve set up an IDS, you should see which kinds of attacks trigger it, and which it ignores. Both blue teamers and red teamers should know what kinds of attacks are better at going unnoticed by an IDS.

From there, learn how to add rules and extra logging sources to thwart your own attacks. If you bypass the IDS, modify the IDS ruleset so your bypass no longer works. Then try to find a new way in. This kind of cat and mouse game in your own network will give you hands-on experience and useful intuitions about pentesting and detecting breaches.

Have fun, and as always, happy hacking!