airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Rawat <ashish.ra...@myntra.com>
Subject Task partitioning using Airflow
Date Thu, 03 Aug 2017 15:43:42 GMT
Hi,

We have a use case where we are running some R/Python based data science models, which execute
over a single node. The execution time of the models is constantly increasing and we are now
planning to split the model training by a partition key and distribute the workload over multiple
machines.

Does Airflow provide some simple way to split a task into multiple tasks, all of which will
work on a specific value of the key.

--
Regards,
Ashish




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message