From commits-return-89310-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Mon Jan 13 14:39:07 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A4A9E18064E for ; Mon, 13 Jan 2020 15:39:06 +0100 (CET) Received: (qmail 73231 invoked by uid 500); 13 Jan 2020 14:39:06 -0000 Mailing-List: contact commits-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list commits@airflow.apache.org Received: (qmail 73222 invoked by uid 99); 13 Jan 2020 14:39:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jan 2020 14:39:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 70188C038A for ; Mon, 13 Jan 2020 14:39:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -112.19 X-Spam-Level: X-Spam-Status: No, score=-112.19 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_DMARC_STATUS=0.01, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id zkYDipAzLLFP for ; Mon, 13 Jan 2020 14:39:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=207.244.88.153; helo=mail.apache.org; envelope-from=jira@apache.org; receiver= Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with SMTP id 0EE117DDF3 for ; Mon, 13 Jan 2020 14:39:02 +0000 (UTC) Received: (qmail 73177 invoked by uid 99); 13 Jan 2020 14:39:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jan 2020 14:39:02 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 91C95E0CB5 for ; Mon, 13 Jan 2020 14:39:01 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 5A9FE780689 for ; Mon, 13 Jan 2020 14:39:00 +0000 (UTC) Date: Mon, 13 Jan 2020 14:39:00 +0000 (UTC) From: "Hamid Mahmood (Jira)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (AIRFLOW-6551) KubernetesExecutor does not create dynamic pods for tasks inside subdag MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-6551?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hamid Mahmood updated AIRFLOW-6551: ----------------------------------- Description:=20 KubernetesExecutor does not create dynamic pods for tasks inside subdag. I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators = inside a main_dag. Following is the hierarchy * main_dag: ** subdagA *** taskA1 *** taskA2 *** taskA3 ** subdagB *** subdagB_1 **** taskB1 **** taskB2 ** task_main1 ** task_main2 ** task_main3 I have set the following parameter =C2=A0 {code:java} AIRFLOW__CORE__EXECUTOR =3D KubernetesExecutor {code} When I run the workflow main_dag, only a single pod is created with name su= bdagA- 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) = run inside that single pod. In theory they should be 3 separate pods for each task inside subdagA.=C2= =A0 Same is the case for subdagB, only a single pod is created to run subdagB, = subdagB_1 and its tasks runs inside that pod.=C2=A0 But the tasks (task_main1, task_main2, task_main3) that are not inside furt= her subdag runs in dynamic pods. =C2=A0 *Following is the log* was: Hi Everyone, KubernetesExecutor does not create dynamic pods for tasks inside subdag. I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators = inside a main_dag. Following is the hierarchy * main_dag: ** subdagA *** taskA1 *** taskA2 *** taskA3 ** subdagB *** subdagB_1 **** taskB1 **** taskB2 ** task_main1 ** task_main2 ** task_main3 I have set the following parameter =C2=A0 {code:java} AIRFLOW__CORE__EXECUTOR =3D KubernetesExecutor {code} When I run the workflow main_dag, only a single pod is created with name su= bdagA- 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) = run inside that single pod. In theory they should be 3 separate pods for each task inside subdagA.=C2= =A0 Same is the case for subdagB, only a single pod is created to run subdagB, = subdagB_1 and its tasks runs inside that pod.=C2=A0 But the tasks (task_main1, task_main2, task_main3) that are not inside furt= her subdag runs in dynamic pods. =C2=A0 *Following is the log* > KubernetesExecutor does not create dynamic pods for tasks inside subdag > ----------------------------------------------------------------------- > > Key: AIRFLOW-6551 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6551 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes > Affects Versions: 1.10.7 > Reporter: Hamid Mahmood > Assignee: Daniel Imberman > Priority: Major > > KubernetesExecutor does not create dynamic pods for tasks inside subdag. > I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operator= s inside a main_dag. Following is the hierarchy > * main_dag: > ** subdagA > *** taskA1 > *** taskA2 > *** taskA3 > ** subdagB > *** subdagB_1 > **** taskB1 > **** taskB2 > ** task_main1 > ** task_main2 > ** task_main3 > I have set the following parameter > =C2=A0 > {code:java} > AIRFLOW__CORE__EXECUTOR =3D KubernetesExecutor > {code} > When I run the workflow main_dag, only a single pod is created with name = subdagA- > 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3= ) run inside that single pod. > In theory they should be 3 separate pods for each task inside subdagA.=C2= =A0 > Same is the case for subdagB, only a single pod is created to run subdagB= , subdagB_1 and its tasks runs inside that pod.=C2=A0 > But the tasks (task_main1, task_main2, task_main3) that are not inside fu= rther subdag runs in dynamic pods. > =C2=A0 > *Following is the log* -- This message was sent by Atlassian Jira (v8.3.4#803005)