Post History
Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally. You can run Airflow in single-node mode, with LocalExecutor, and w...
Answer
#3: Post edited
by
matthewsnyder
·
2023-12-08T05:05:40Z (5 months ago)
- Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally.
- You can run Airflow in single-node mode, with `LocalExecutor`, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot.
- In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using `BashOperator` and `PythonOperator` to run code respectively. The DAGs can be versioned with Git in various ways.
- Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs.
- The biggest issues would be:
- * Resource consumption is high
* It is much more complex to set up than cron or systemd
- Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally.
- You can run Airflow in single-node mode, with `LocalExecutor`, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot.
- In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using `BashOperator` and `PythonOperator` to run code respectively. The DAGs can be versioned with Git in various ways.
- Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs.
- The biggest issues would be:
- * Resource consumption is high
- * It is much more complex to set up than cron or systemd
- * If you do use a container, you will then have to find a way for the container to run commands on the host
#2: Post edited
by
matthewsnyder
·
2023-12-08T05:05:08Z (5 months ago)
- Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally.
- You can run Airflow in single-node mode, with `LocalExecutor`, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot.
- In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using `BashOperator` and `PythonOperator` to run code respectively. The DAGs can be versioned with Git in various ways.
- Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs.
- The biggest issues would be:
* Resource consumption would be harder- * It is much more complex to set up than cron or systemd
- Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally.
- You can run Airflow in single-node mode, with `LocalExecutor`, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot.
- In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using `BashOperator` and `PythonOperator` to run code respectively. The DAGs can be versioned with Git in various ways.
- Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs.
- The biggest issues would be:
- * Resource consumption is high
- * It is much more complex to set up than cron or systemd
#1: Initial revision
by
matthewsnyder
·
2023-12-07T20:02:41Z (5 months ago)
Airflow is a distributed workflow manager intended for complex cloud computing use. However, it can be used to run tasks locally. You can run Airflow in single-node mode, with `LocalExecutor`, and wrap the whole thing in a single Docker container. The Docker container can be configured to start automatically at boot. In Airflow, tasks are modeled as dependency DAGs expressed with a Python API. Each DAG is a Python file, so knowing Python is necessary here. Airflow can do many things, but for this question you would mostly be using `BashOperator` and `PythonOperator` to run code respectively. The DAGs can be versioned with Git in various ways. Airflow has very powerful features for complex scheduling, tracking status and logs from past runs and dealing with missed/failed runs. The biggest issues would be: * Resource consumption would be harder * It is much more complex to set up than cron or systemd