airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: MODERATE for dev@airflow.incubator.apache.org
Date Fri, 20 May 2016 16:35:08 GMT
Pavlo,
Please follow the instructions in the  Airflow Contributor Guide
<https://cwiki.apache.org/confluence/display/AIRFLOW/Contributor+Guide> to
create a mailing list account on dev and commits.

This will avoid us from having to accept/reject your individual emails.
When you email dev-subscribe@airflow.incubator.apache.org, a bot will take
care of your subscription.

-s

On Thu, May 19, 2016 at 3:07 PM, <
dev-reject-1463670472.76373.oebhcfdbhheddaapmdja@airflow.incubator.apache.org
> wrote:

>
> To approve:
>
> dev-accept-1463670472.76373.oebhcfdbhheddaapmdja@airflow.incubator.apache.org
> To reject:
>
> dev-reject-1463670472.76373.oebhcfdbhheddaapmdja@airflow.incubator.apache.org
> To give a reason to reject:
> %%% Start comment
> %%% End comment
>
>
>
> ---------- Forwarded message ----------
> From: "Ryabchuk, Pavlo" <ext-pavlo.ryabchuk@here.com>
> To: "dev@airflow.incubator.apache.org" <dev@airflow.incubator.apache.org>
> Cc:
> Date: Thu, 19 May 2016 15:07:42 +0000
> Subject: Using CeleryExecutor to distribute SubDags, but running SubDAG
> tasks locally one by one afterwards
>
> Hello,
>
>
>
> I am trying to fully use Airflow analytics and trying to make my tasks as
> granular as possible still having the benefit of CeleryExecutor present.
>
> In general I want to make my DAG to consist of 100+ Subgags which are
> actually distributed by Celery, but I want to have subdag tasks all
> executed on same worker instance.
>
> SubDAG is in general this: copyData data from S3 to instance -> run
> calculation -> copy result to S3. The reason I want to split it into 3
> tasks is to have ability to measure pure calculation time and aply SLA to
> it and also get better statisticts on copy operations.
>
> So the main question is how to execute tasks of SubDAG one after another
> on same instance?
>
>
>
> I’ve came across this issue/workaround here
> https://issues.apache.org/jira/browse/AIRFLOW-74, but I believe it won’t
> solve my issue.
>
> If it is not supported and I am not missing some magic configuration J
> but still could be implemented with relatively small effort – I am in :)
>
>
>
> Best,
>
> Paul
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message