airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niranda Perera <niranda...@cse.mrt.ac.lk>
Subject Re: Selectively send jobs to executors - executor pooling
Date Tue, 16 May 2017 05:48:54 GMT
Hi Max,

Thank you very much for the clarification. I revisited the docs and I
believe queues would address my requirement. I will explain you my
requirement so that I could verify if queues are the answer for this.

Say, my pipeline is as follows
A(linux) --> B(linux) --> D (win) --> E (win) --> end
             --> C(linux) -----------------> F (win) ___|

So, I want to make sure that these tasks reach the correct worker which has
the correct env (linux/ win).
Assumption: Airflow works on windows and linux :-)

my idea is to have 2 queues win_q and linux_q and make windows/ linux
workers to listen to the relevant queue only. Is this possible with the
current provisions of airflow?

Best regards
Nira


Niranda Perera
Research Assistant
Dept of CSE, University of Moratuwa
niranda.17@cse.mrt.ac.lk
+94 71 554 8430 <+94%2071%20554%208430>
https://lk.linkedin.com/in/niranda

On Tue, May 16, 2017 at 9:48 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> I think you might be confusing some of the terms & concepts.
>
> Tasks can be wired to arbitrary queues, which represent one or many workers
> of a certain type, potentially with a specific configuration and
> provisioning.
> https://airflow.incubator.apache.org/concepts.html#queues
>
> Pools are a way to limit concurrency for a set of tasks
> https://airflow.incubator.apache.org/concepts.html#pools
>
> Executor is an interface that defines how tasks are triggered and
> eventually using some sort of remote execution backend. Currently mostly
> the LocalExecutor and CeleryExecutor are used by the community, but we
> could have implementation for platforms like Mesos, Kubernetes or Yarn.
>
> Please clarify your question if this bit of information doesn't answer it.
>
> Max
>
> On Mon, May 15, 2017 at 6:59 PM, Niranda Perera <niranda.17@cse.mrt.ac.lk>
> wrote:
>
> > Hi all,
> >
> > I am very new to Airflow. I am looking for a particular feature in
> airflow,
> > but I could not find out whether airflow supports it.
> >
> > What I am looking for is, executor pooling. i.e. assigning a set of jobs
> to
> > a particular pool of executors.
> >
> > Is this supported in airflow, or would airflow submit jobs to a common
> pool
> > of executors?
> >
> > Best regards
> >
> > Niranda Perera
> > Research Assistant
> > Dept of CSE, University of Moratuwa
> > niranda.17@cse.mrt.ac.lk
> > +94 71 554 8430
> > https://lk.linkedin.com/in/niranda
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message