Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Comments on How to detect if a Linux Docker host has had unwelcome guests?

Parent

How to detect if a Linux Docker host has had unwelcome guests?

+3
−0

I plan to set up a Raspberry Pi to host some dockerized apps, for it to be accessible to the Internet through a remote proxy, and since I'm no sysadmin, I'm trying to come up with a suitable policy for security and backups.

Besides common security measures such keeping stuff up-to-date, using a firewall, SSH keys, I want to periodically check the system configuration to see whether any unwelcome guests tampered with the Pi, so I can reinstall everything and be up and running with minimum hassle.

For this, I have a backup of partitions of my base system (boot on MMC and root on an attached HDD), and then make periodic remote backups of the following:

  1. all the Docker volume directories
  2. MariaDB dumps for every database
  3. All of /etc
  4. the output of apt list --installed
  5. the output of export for the root user
  6. Some or all of /var, /usr/, /bin, '/sbin and /boot, not sure yet.

The first two are the actual backup, whereas 4-6 I intend to get check whether they have changed in unexpected ways that would justify a reinstall.

My questions are then:

  1. Is there a better way to monitor my system for changes?
  2. Which of item 6 make sense to monitor? Am I missing anything else?

Thanks in advance.

History
Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

0 comment threads

Post
+4
−0

You can't expect an attacker to abide by any particular rulebook.

In fact, if they did, then the defenders' job would be so much easier. That's why attackers don't do it.

Remember the adage: a defender has to defend against every possible attack everywhere, but an attacker only has to find one spot which is undefended from one attack in order to succeed.

Somewhere around 20 years ago, I had a Linux system compromised. I only realized that something was amiss when I saw a .. entry in the output of ls /dev or something like that. Turned out that the attacker had dropped their stuff into a directory /dev/.. / (that's dot-dot-space). None of what you discuss would have caught that.

Modern attacks are far more sophisticated than that.

More generally, you can't rely on a compromised system to tell you that it has been compromised, precisely because it has been compromised. An attacker that is in a position to tamper with files in /bin, or drop stuff into /dev, could just as well be poking around in /lib or modify /sbin/init in whatever manner they choose. And if they can cause changes to, say, anything in /etc/systemd/system or /etc/ld.so.conf.d, then really, all bets are off: they own the system and you are just along for the ride.

This leads to two main conclusions:

  • Don't have a(n even potentially) vulnerable system push backups elsewhere. Any attacker who has penetrated that system can then directly affect the backups as well. Always pull backups onto a secured system, in the most restricted manner possible, and set up the backed-up host such that the backup server can't push files onto it. (The latter is often overlooked in suggestions online. With rsync over SSH, for example, you might want to restrict to rsync --server --sender on the backed-up host.)

  • Don't try to guess what an attacker might do. Instead, assume that they will think of something that you haven't thought of, and monitor everything for changes except things that you know are legitimate changes. This is the approach taken by venerable tools such as logcheck and tripwire: instead of looking for specific hostile activity, assume that everything is at least potentially hostile unless you know that it's friendly.

Also, as somewhat of a corollary to the second point:

  • Don't try to analyze on a system whether that system has been somehow compromised. While there might be clues that something isn't quite right, any moderately competently performed attack will make actual determination that an attack has occurred anywhere from difficult to impossible. You have to do such analysis on a system that you can know is not tampered with.

And, again an old adage:

  • If you realize that the system has been compromised, nuke it from orbit and reinstall from scratch from known good installation media. It's the only way to be sure. If you restore from backups, then you absolutely must do so selectively and only after careful analysis of what you are restoring to ensure that you aren't reintroducing a backdoor for the attacker.
History
Why does this post require moderator attention?
You might want to add some details to your flag.

1 comment thread

Thanks so much for the thorough explanation and the sound advise. I was kind of aware that *some* stu... (4 comments)
Thanks so much for the thorough explanation and the sound advise. I was kind of aware that *some* stu...
nnlei‭ wrote over 2 years ago · edited over 2 years ago

Thanks so much for the thorough explanation and the sound advise. I was kind of aware that some stuff would escape me, just not that it was almost everything, and most important stuff :) My idea was to pull backups from my personal PC with rsync, but now I'm more inclined to do offline backups and check the system from a brand new Debian installation in a box that's not connected to any network.

nnlei‭ wrote over 2 years ago

Also, given my current skill level, I guess it would make sense for me to reinstall the system from scratch periodically, from media I'm sure to be clean, even if nothing seems off. That way I wouldn't need to check most stuff, only the app data and some individual files.

Canina‭ wrote over 2 years ago

nnlei‭ Reinstalling from scratch just because a certain amount of time has passed seems on the one hand quite heavy-handed, and on the other not quite enough (imagine if the system is compromised immediately after the reinstallation through a vulnerability which has been patched but for which you haven't had time to apply the patch). If you do go that route, at a minimum carefully document, and ideally automate, the setup process from a fresh OS installation right after first reboot to an up-to-date system with the Dockerized apps up, running and appropriately accessible. (Doing so will probably teach you a lot about shell scripting!) Also be sure to ensure that some manner of update automation is active, whether by cron-apt, unattended-upgrades, or something else.

nnlei‭ wrote over 2 years ago · edited over 2 years ago

Canina‭ Thanks again. I am currently figuring out how to best automate this :) I was thinking of running apt update && apt full-upgrade first thing on the first boot, and then daily through cron as you mentioned (my base image starts with ufw enabled with all ports closed). Is there anything I can do to prevent my system from getting compromised before that first update takes place, besides keeping my OpenWrt router up to date?