airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hamid Mahmood (Jira)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-6551) KubernetesExecutor does not create dynamic pods for tasks inside subdag
Date Mon, 13 Jan 2020 14:39:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hamid Mahmood updated AIRFLOW-6551:
-----------------------------------
    Description: 
KubernetesExecutor does not create dynamic pods for tasks inside subdag.

I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag.
Following is the hierarchy
 * main_dag:
 ** subdagA
 *** taskA1
 *** taskA2
 *** taskA3
 ** subdagB
 *** subdagB_1
 **** taskB1
 **** taskB2
 ** task_main1
 ** task_main2
 ** task_main3

I have set the following parameter

 
{code:java}
AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
{code}
When I run the workflow main_dag, only a single pod is created with name subdagA-

309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside that
single pod.

In theory they should be 3 separate pods for each task inside subdagA. 

Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1 and its
tasks runs inside that pod. 

But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag runs
in dynamic pods.

 

*Following is the log*

  was:
Hi Everyone,

KubernetesExecutor does not create dynamic pods for tasks inside subdag.

I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag.
Following is the hierarchy
 * main_dag:
 ** subdagA
 *** taskA1
 *** taskA2
 *** taskA3
 ** subdagB
 *** subdagB_1
 **** taskB1
 **** taskB2
 ** task_main1
 ** task_main2
 ** task_main3

I have set the following parameter

 
{code:java}
AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
{code}
When I run the workflow main_dag, only a single pod is created with name subdagA-

309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside that
single pod.

In theory they should be 3 separate pods for each task inside subdagA. 

Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1 and its
tasks runs inside that pod. 

But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag runs
in dynamic pods.

 



*Following is the log*


> KubernetesExecutor does not create dynamic pods for tasks inside subdag
> -----------------------------------------------------------------------
>
>                 Key: AIRFLOW-6551
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6551
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executor-kubernetes
>    Affects Versions: 1.10.7
>            Reporter: Hamid Mahmood
>            Assignee: Daniel Imberman
>            Priority: Major
>
> KubernetesExecutor does not create dynamic pods for tasks inside subdag.
> I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag.
Following is the hierarchy
>  * main_dag:
>  ** subdagA
>  *** taskA1
>  *** taskA2
>  *** taskA3
>  ** subdagB
>  *** subdagB_1
>  **** taskB1
>  **** taskB2
>  ** task_main1
>  ** task_main2
>  ** task_main3
> I have set the following parameter
>  
> {code:java}
> AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
> {code}
> When I run the workflow main_dag, only a single pod is created with name subdagA-
> 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside
that single pod.
> In theory they should be 3 separate pods for each task inside subdagA. 
> Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1
and its tasks runs inside that pod. 
> But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag
runs in dynamic pods.
>  
> *Following is the log*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message