There are two ways you can run airflow as a mesos framework:
MesosExecutor
allows you to schedule airflow tasks on a Mesos cluster.
For this to work, you need a running mesos cluster and you must perform the following
steps -
airflow.cfg
.airflow.cfg
to point executor parameter to
MesosExecutor and provide related Mesos settings.airflow.cfg
from
Airflow server (so that it uses same sql alchemy connection).airflow serve_logs
airflow scheduler -p
Note: We need -p parameter to pickle the DAGs.
You can now see the airflow framework and corresponding tasks in mesos UI. The logs for airflow tasks can be seen in airflow UI as usual.
For more information about mesos, refer to mesos documentation. For any queries/bugs on MesosExecutor, please contact @kapil-malik.
This gist contains all files and configuration changes necessary to achieve the following:
We recommend taking advantage of docker’s multi stage builds in order to achieve this. We have one Dockerfile that defines building a specific version of mesos from source (Dockerfile-mesos), in order to create the python eggs. In the airflow Dockerfile (Dockerfile-airflow) we copy the python eggs from the mesos image.
airflow.cfg
.The configuration block remains the same as the default airflow configuration (default_airflow.cfg), but has the addition of an optiondocker_image_slave
. This should be set to the name of the image you would like mesos to use when running airflow tasks. Make sure you have the proper configuration of the DNS record for your mesos master and any sort of authorization if any exists.
airflow.cfg
to point the executor parameter to
MesosExecutor (executor = SequentialExecutor).docker_image_slave
.The rest is up to you and how you want to work with a dockerized airflow configuration.