airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
Date Fri, 13 Sep 2019 08:37:23 GMT
clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
URL: https://github.com/apache/airflow/pull/5498#issuecomment-531151925
 
 
   Tried on master today.
   I trigger main DAG, all sub-DAG runs start running before their SubDagOperator is reached.
   
   - Is it possible to make e.g. TASK3.TASK1 depend on the completion of TASK2?
    - Can I make a (subclassed) SubDagOperator xcom_push something on execute that the head
of the sub-DAG can wait for?
    - Can I make SubDagOperator not create a DagRun when its DagRun is created?
   
   Other workarounds I'm considering are:
    - Stay on 1.10.5, give SubDagOperators their own big worker queue where they can sit and
wait without obstructing sensors.
    - Increase number of workers without queue segregation.
    - Use disconnected DAGs and TriggerDagRunOperator instead of SubDagOperator.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message