airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Rawat <>
Subject Task partitioning using Airflow
Date Thu, 03 Aug 2017 15:43:42 GMT

We have a use case where we are running some R/Python based data science models, which execute
over a single node. The execution time of the models is constantly increasing and we are now
planning to split the model training by a partition key and distribute the workload over multiple

Does Airflow provide some simple way to split a task into multiple tasks, all of which will
work on a specific value of the key.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message