Modern and practical way to schedule tasks on a Linux machine that is not always on
What is a modern and practical way to handle scheduling tasks on a Linux desktop?
- Modern means basic features should not rely on obscure or convoluted commands and standards
- Practical means you shouldn't jump through too many hoops to do basic things, it should be usable for 10-100 tasks
- I specify Linux desktop to emphasize that the machine will not be always on, and the approach should still work without that expectation
To be more specific, I want to schedule various things like run a script to check my email, check some sites I follow for updates, check for system updates, do backups. The schedule will usually be a cron-style date time period. When a job run is missed because the computer was not on, I will sometimes want it to be "caught up with" next time it is on, and sometimes to just be ignored, depending on the exact task.
There are some options that I know about, but in my opinion none are ideal. I will post them as separate answers to help add context.
3 answers
The standard way to schedule tasks in most Linux systems nowadays is to use systemd timers.
This requires writing a systemd timer config file, which has syntax similar to INI. This is summarized in https://wiki.archlinux.org/title/Systemd/Timers and covered in detail in systemd docs.
The timers can be placed in various directories, notably one of these is under ~/.config/systemd/user which is convenient for versioning it as dotfiles. However, this is very limited, because it does not allow creating subfolder.
The benefit of systemd timers is that they are currently ubiquitous, and systemd has powerful features for keeping track of logs and fine tuning the exact schedule. The drawback is that the timer configs end up being a bit complex, and systemd itself is complex and not that easy to use.
Historically, cron was the main way to schedule tasks on Linux. Briefly, there will be some file like /etc/crontab which will contain one line for each task. The line starts with a schedule string and a shell style command.
The schedule string specifies on which minutes, hours, days and months the task should run. When the cron daemon is running, whenever it detects that a time matching the schedule is reached, it executes the command.
When a cron schedule is missed, it's missed. Vanilla cron only cares about what happens when the machine is on. There are extensions like anacron which deal with this edge case.
Systemd timers were meant to replace cron, so many distros now consider cron obsolete and don't install it by default. However, there are programs like cronie and fcron which you can install and obtain the same functionality.
The caveats with cron are:
- When commands fail (cron looks at the exit code) they "send an email". This is not an actual email over the internet, but using an ancient internal mail functionality of Unix, to send it to your Linux user account's "mailbox". These days most people do not understand how this works and get confused by it.
- There is little support for dependencies between tasks, you will have to bake that into your script instead.
- AFAIK crontabs require sudo to edit, and you're not even supposed to edit them directly but use
crontab -e. Sucks if you were hoping to replace it with a symlink to your dotfiles repo.
0 comment threads
Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally.
You can run Airflow in single-node mode, with LocalExecutor, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot.
In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using BashOperator and PythonOperator to run code respectively. The DAGs can be versioned with Git in various ways.
Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs.
The biggest issues would be:
- Resource consumption is high
- It is much more complex to set up than cron or systemd
- If you do use a container, you will then have to find a way for the container to run commands on the host
0 comment threads
0 comment threads