Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
References: <0DAC7CB2-E1AB-4336-ACA1-A4F85517F36C@myntra.com> <63890AFD-6D26-46B3-BEF7-378FB29FD7B4@myntra.com>
In-Reply-To: <63890AFD-6D26-46B3-BEF7-378FB29FD7B4@myntra.com>
From: George Leslie-Waksman <george@cloverhealth.com.INVALID>
Date: Wed, 09 Aug 2017 17:10:39 +0000
Message-ID: <CANcybpAk58hw=cXw=VJgWc=MdnvuYw3QzyG_2xx8c8MVBLgybw@mail.gmail.com>
Subject: Re: Task partitioning using Airflow
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a113cd73c5de3cd055655298b"
archived-at: Wed, 09 Aug 2017 17:10:56 -0000

--001a113cd73c5de3cd055655298b
Content-Type: text/plain; charset="UTF-8"

Airflow is best for situations where you want to run different tasks that
depend on each other or process data that arrives over time. If your goal
is to take a large dataset, split it up, and process chunks of it, there
are probably other tools better suited to your purpose.

Off the top of my head, you might consider Dask:
https://dask.pydata.org/en/latest/ or directly using Celery:
http://www.celeryproject.org/

--George

On Wed, Aug 9, 2017 at 9:52 AM Ashish Rawat <ashish.rawat@myntra.com> wrote:

> Hi - Can anyone please provide some pointers for this use case over
> Airflow?
>
> --
> Regards,
> Ashish
>
>
>
> > On 03-Aug-2017, at 9:13 PM, Ashish Rawat <ashish.rawat@myntra.com>
> wrote:
> >
> > Hi,
> >
> > We have a use case where we are running some R/Python based data science
> models, which execute over a single node. The execution time of the models
> is constantly increasing and we are now planning to split the model
> training by a partition key and distribute the workload over multiple
> machines.
> >
> > Does Airflow provide some simple way to split a task into multiple
> tasks, all of which will work on a specific value of the key.
> >
> > --
> > Regards,
> > Ashish
> >
> >
> >
>
>

--001a113cd73c5de3cd055655298b--