Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAB4JZV+NUeuSaB43+4ebSJ+dyHAxaaTXZ5W0OX1XHtJX6PfMVA@mail.gmail.com>
References: <CAB4JZV+NUeuSaB43+4ebSJ+dyHAxaaTXZ5W0OX1XHtJX6PfMVA@mail.gmail.com>
From: Victor Monteiro <victor.monteiro@ubee.in>
Date: Thu, 3 Aug 2017 14:24:52 -0300
Message-ID: <CANSE2SMDdKY7CFmg9mottTNS-_=UocrzwU6TSBdHO=2KoAxmtQ@mail.gmail.com>
Subject: Re: Per-task resources with Mesos
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="f403043ed3248c4f9b0555dca80b"
archived-at: Thu, 03 Aug 2017 17:25:03 -0000

--f403043ed3248c4f9b0555dca80b
Content-Type: text/plain; charset="UTF-8"

Hi Stefano, have you read about queues? Airflow has this concept and I
think you can decide for which queue a task should go. By doing this and
integrating it with mesos, I believe you can make a mesos cluster with more
resources to get tasks from a certain queue specific for heavy computations.

Maybe this can solve your problem (not sure) :D

2017-08-03 4:34 GMT-03:00 Stefano Baghino <stefano.baghino@teralytics.ch>:

> Hi everyone,
>
> I'm investigating the possibility for our organization to use Airflow for
> workflow management.
>
> Some requirements on our side regard resource management, and in particular
> the possibility for the system to run tasks on top of Apache Mesos. Airflow
> partially satisfies our requirements in that regard, meaning that after
> having a look at the docs and code, it appears to me (correct me if I'm
> wrong) that resources are determined for the whole system (via
> configuration) and cannot be determined on a per-task basis. We'd need this
> because some of our jobs are quite lightweight while others may require a
> lot of resources, making it a "one-size-fits-all" configuration quite
> wasteful.
>
> I had a look at the AirflowMesosScheduler and MesosExecutor and thought it
> would be nice to add this feature and perhaps I can add it myself. What I
> would need is some guidance on how to make this fit into the overall system
> design: is there an established way to explicitly ask for resources for a
> specific task in the DAG? If not, what could be a possible way to introduce
> it? And if this reveals itself to be outside of the scope of Airflow, how
> do you think I can make it meet our requirement?
>
> Thanks in advance.
>
> P.S.: if by any chance some of you are on the Mesos mailing list as well,
> you may know that I'm having issues in making Airflow run successfully
> using Mesos due to missing Python packages. I'm not sure whether this
> mailing list is an appropriate place for users to get help. If so, I could
> probably share that post here as well. Thanks!
>
> --
> Stefano Baghino | TERALYTICS
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41 43 508 24 57
> email: stefano.baghino@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if
> you think that it may not be intended for you and delete it immediately.
>

--f403043ed3248c4f9b0555dca80b--