How to detect if a Linux Docker host has had unwelcome guests?
I plan to set up a Raspberry Pi to host some dockerized apps, for it to be accessible to the Internet through a remote proxy, and since I'm no sysadmin, I'm trying to come up with a suitable policy for security and backups.
Besides common security measures such keeping stuff up-to-date, using a firewall, SSH keys, I want to periodically check the system configuration to see whether any unwelcome guests tampered with the Pi, so I can reinstall everything and be up and running with minimum hassle.
For this, I have a backup of partitions of my base system (boot on MMC and root on an attached HDD), and then make periodic remote backups of the following:
- all the Docker volume directories
- MariaDB dumps for every database
- All of
/etc - the output of
apt list --installed - the output of
exportfor the root user - Some or all of
/var,/usr/,/bin, '/sbinand/boot, not sure yet.
The first two are the actual backup, whereas 4-6 I intend to get check whether they have changed in unexpected ways that would justify a reinstall.
My questions are then:
- Is there a better way to monitor my system for changes?
- Which of item 6 make sense to monitor? Am I missing anything else?
Thanks in advance.
1 answer
You can't expect an attacker to abide by any particular rulebook.
In fact, if they did, then the defenders' job would be so much easier. That's why attackers don't do it.
Remember the adage: a defender has to defend against every possible attack everywhere, but an attacker only has to find one spot which is undefended from one attack in order to succeed.
Somewhere around 20 years ago, I had a Linux system compromised. I only realized that something was amiss when I saw a .. entry in the output of ls /dev or something like that. Turned out that the attacker had dropped their stuff into a directory /dev/.. / (that's dot-dot-space). None of what you discuss would have caught that.
Modern attacks are far more sophisticated than that.
More generally, you can't rely on a compromised system to tell you that it has been compromised, precisely because it has been compromised. An attacker that is in a position to tamper with files in /bin, or drop stuff into /dev, could just as well be poking around in /lib or modify /sbin/init in whatever manner they choose. And if they can cause changes to, say, anything in /etc/systemd/system or /etc/ld.so.conf.d, then really, all bets are off: they own the system and you are just along for the ride.
This leads to two main conclusions:
-
Don't have a(n even potentially) vulnerable system push backups elsewhere. Any attacker who has penetrated that system can then directly affect the backups as well. Always pull backups onto a secured system, in the most restricted manner possible, and set up the backed-up host such that the backup server can't push files onto it. (The latter is often overlooked in suggestions online. With rsync over SSH, for example, you might want to restrict to
rsync --server --senderon the backed-up host.) -
Don't try to guess what an attacker might do. Instead, assume that they will think of something that you haven't thought of, and monitor everything for changes except things that you know are legitimate changes. This is the approach taken by venerable tools such as logcheck and tripwire: instead of looking for specific hostile activity, assume that everything is at least potentially hostile unless you know that it's friendly.
Also, as somewhat of a corollary to the second point:
- Don't try to analyze on a system whether that system has been somehow compromised. While there might be clues that something isn't quite right, any moderately competently performed attack will make actual determination that an attack has occurred anywhere from difficult to impossible. You have to do such analysis on a system that you can know is not tampered with.
And, again an old adage:
- If you realize that the system has been compromised, nuke it from orbit and reinstall from scratch from known good installation media. It's the only way to be sure. If you restore from backups, then you absolutely must do so selectively and only after careful analysis of what you are restoring to ensure that you aren't reintroducing a backdoor for the attacker.
1 comment thread
0 comment threads