Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAF+3fpOkpG2Qx3Lvwg14wObDM=C_dwxH-4puDg1L8=hKpCjVvA@mail.gmail.com>
References: <CA+1xF3oECsw3F6o+mJ-qKhOLpQMOxk3ef746Etoh1a=kpnG7VA@mail.gmail.com>
 <CAHEEp7X6XNoii5zZA18cdh=ox+8mwLiXkKnzzdgxdosZ=takNg@mail.gmail.com>
 <CA+1xF3owcF+evhKtu09jcdK4eTbhaZ_FBiH+r_Qy15yV7+FvGQ@mail.gmail.com>
 <CAHEEp7UHjEt4Kmdvvn7FOEJJsEXTJvfP0_cCf5mOCd7cb8=omA@mail.gmail.com> <CAF+3fpOkpG2Qx3Lvwg14wObDM=C_dwxH-4puDg1L8=hKpCjVvA@mail.gmail.com>
From: Niranda Perera <niranda.17@cse.mrt.ac.lk>
Date: Tue, 16 May 2017 12:09:30 +0530
Message-ID: <CA+1xF3o1jqmkHgQxY=O8oy8RhTT1wSjo2CmmM2m+_ZfZQCP0YA@mail.gmail.com>
Subject: Re: Selectively send jobs to executors - executor pooling
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a11e67daa06bcd8054f9e70ae"
archived-at: Tue, 16 May 2017 06:39:52 -0000

--001a11e67daa06bcd8054f9e70ae
Content-Type: text/plain; charset="UTF-8"

Hi Arthur,

so, IIUC, I should still be able to monitor the windows celery worker from
the airflow master node (theoretically), isn't it? I am not planning to use
windows for the master. the requirement is that some apps in the workflow
are only compatible with windows.

If this works, I will surely update the mailing list :-)

Best
Nira


Best regards

Niranda Perera
Research Assistant
Dept of CSE, University of Moratuwa
niranda.17@cse.mrt.ac.lk
+94 71 554 8430
https://lk.linkedin.com/in/niranda

On Tue, May 16, 2017 at 12:04 PM, Arthur Wiedmer <arthur.wiedmer@gmail.com>
wrote:

> I think that last we checked, Gunicorn which powers the webserver was not
> compatible with Windows, but this might be changing now that Windows is
> exposing a Unix interface of sorts.
>
> The worker should be fine though, as Celery runs on Windows.
>
> Nira, if it works for you, please let us know on the mailing list :)
>
> Best,
> Arthur
>
> On Mon, May 15, 2017 at 11:24 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Yes that's what queues are for. Though I doubt Airflow works on Windows,
> I
> > haven't heard of anyone doing this.
> >
> > Max
> >
> > On Mon, May 15, 2017 at 10:48 PM, Niranda Perera <
> niranda.17@cse.mrt.ac.lk
> > >
> > wrote:
> >
> > > Hi Max,
> > >
> > > Thank you very much for the clarification. I revisited the docs and I
> > > believe queues would address my requirement. I will explain you my
> > > requirement so that I could verify if queues are the answer for this.
> > >
> > > Say, my pipeline is as follows
> > > A(linux) --> B(linux) --> D (win) --> E (win) --> end
> > >              --> C(linux) -----------------> F (win) ___|
> > >
> > > So, I want to make sure that these tasks reach the correct worker which
> > has
> > > the correct env (linux/ win).
> > > Assumption: Airflow works on windows and linux :-)
> > >
> > > my idea is to have 2 queues win_q and linux_q and make windows/ linux
> > > workers to listen to the relevant queue only. Is this possible with the
> > > current provisions of airflow?
> > >
> > > Best regards
> > > Nira
> > >
> > >
> > > Niranda Perera
> > > Research Assistant
> > > Dept of CSE, University of Moratuwa
> > > niranda.17@cse.mrt.ac.lk
> > > +94 71 554 8430 <+94%2071%20554%208430>
> > > https://lk.linkedin.com/in/niranda
> > >
> > > On Tue, May 16, 2017 at 9:48 AM, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > I think you might be confusing some of the terms & concepts.
> > > >
> > > > Tasks can be wired to arbitrary queues, which represent one or many
> > > workers
> > > > of a certain type, potentially with a specific configuration and
> > > > provisioning.
> > > > https://airflow.incubator.apache.org/concepts.html#queues
> > > >
> > > > Pools are a way to limit concurrency for a set of tasks
> > > > https://airflow.incubator.apache.org/concepts.html#pools
> > > >
> > > > Executor is an interface that defines how tasks are triggered and
> > > > eventually using some sort of remote execution backend. Currently
> > mostly
> > > > the LocalExecutor and CeleryExecutor are used by the community, but
> we
> > > > could have implementation for platforms like Mesos, Kubernetes or
> Yarn.
> > > >
> > > > Please clarify your question if this bit of information doesn't
> answer
> > > it.
> > > >
> > > > Max
> > > >
> > > > On Mon, May 15, 2017 at 6:59 PM, Niranda Perera <
> > > niranda.17@cse.mrt.ac.lk>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am very new to Airflow. I am looking for a particular feature in
> > > > airflow,
> > > > > but I could not find out whether airflow supports it.
> > > > >
> > > > > What I am looking for is, executor pooling. i.e. assigning a set of
> > > jobs
> > > > to
> > > > > a particular pool of executors.
> > > > >
> > > > > Is this supported in airflow, or would airflow submit jobs to a
> > common
> > > > pool
> > > > > of executors?
> > > > >
> > > > > Best regards
> > > > >
> > > > > Niranda Perera
> > > > > Research Assistant
> > > > > Dept of CSE, University of Moratuwa
> > > > > niranda.17@cse.mrt.ac.lk
> > > > > +94 71 554 8430
> > > > > https://lk.linkedin.com/in/niranda
> > > > >
> > > >
> > >
> >
>

--001a11e67daa06bcd8054f9e70ae--