airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Victor Monteiro <>
Subject Re: Per-task resources with Mesos
Date Thu, 03 Aug 2017 17:24:52 GMT
Hi Stefano, have you read about queues? Airflow has this concept and I
think you can decide for which queue a task should go. By doing this and
integrating it with mesos, I believe you can make a mesos cluster with more
resources to get tasks from a certain queue specific for heavy computations.

Maybe this can solve your problem (not sure) :D

2017-08-03 4:34 GMT-03:00 Stefano Baghino <>:

> Hi everyone,
> I'm investigating the possibility for our organization to use Airflow for
> workflow management.
> Some requirements on our side regard resource management, and in particular
> the possibility for the system to run tasks on top of Apache Mesos. Airflow
> partially satisfies our requirements in that regard, meaning that after
> having a look at the docs and code, it appears to me (correct me if I'm
> wrong) that resources are determined for the whole system (via
> configuration) and cannot be determined on a per-task basis. We'd need this
> because some of our jobs are quite lightweight while others may require a
> lot of resources, making it a "one-size-fits-all" configuration quite
> wasteful.
> I had a look at the AirflowMesosScheduler and MesosExecutor and thought it
> would be nice to add this feature and perhaps I can add it myself. What I
> would need is some guidance on how to make this fit into the overall system
> design: is there an established way to explicitly ask for resources for a
> specific task in the DAG? If not, what could be a possible way to introduce
> it? And if this reveals itself to be outside of the scope of Airflow, how
> do you think I can make it meet our requirement?
> Thanks in advance.
> P.S.: if by any chance some of you are on the Mesos mailing list as well,
> you may know that I'm having issues in making Airflow run successfully
> using Mesos due to missing Python packages. I'm not sure whether this
> mailing list is an appropriate place for users to get help. If so, I could
> probably share that post here as well. Thanks!
> --
> Stefano Baghino | TERALYTICS
> *software engineer*
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41 43 508 24 57
> email:
> Company registration number: CH- | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if
> you think that it may not be intended for you and delete it immediately.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message