airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: feature suggestion
Date Wed, 20 Jun 2018 18:59:56 GMT
The new lineage functionality (while in very early states) can do this for you.

B.

> On 20 Jun 2018, at 19:46, Andrew Harmon <andrewhharmon@gmail.com> wrote:
> 
> Hello,
> 
> I wanted to submit a feature suggestion. I have a scenario where I'd like
> to create a subdag where the number of tasks in it are dependent on a
> previous task. Ideally, this parameter could  be passed in via xcom. For
> example:
> 
> Task A does something that generates 10 files. Task B is a subdag that at
> runtime, creates 10 tasks within the sub dag. Each task will process one of
> the 10 files. That way the processing of the 10 files can run in parallel.
> Task C will then start when all tasks in the subdag are complete.
> 
> My particular scenario is to process the results of a BigQuery export to
> GCS. The export creates n number of files and you don't know that number
> until the export is done running.
> 
> Thanks,
> Andrew
> -- 
> Andrew Harmon
> (202) 615-6433


Mime
View raw message