From dev-return-8016-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed Apr 10 06:41:47 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 63956180626 for ; Wed, 10 Apr 2019 08:41:47 +0200 (CEST) Received: (qmail 42698 invoked by uid 500); 10 Apr 2019 06:41:46 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 42686 invoked by uid 99); 10 Apr 2019 06:41:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Apr 2019 06:41:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A1810C2925 for ; Wed, 10 Apr 2019 06:41:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=slack-corp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xNAPx__z5PAX for ; Wed, 10 Apr 2019 06:41:44 +0000 (UTC) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id F121C6250B for ; Wed, 10 Apr 2019 06:41:43 +0000 (UTC) Received: by mail-io1-f41.google.com with SMTP id p23so1101313iol.13 for ; Tue, 09 Apr 2019 23:41:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=DE3+nr/tFyTFw0STUZpwxF1lVY3MsZv5s1uT/O/rnbM=; b=KMo8ojHqjo3wDrkRlP1q84c3Yf144rk3kdDwBOPAc51kZCzTAhkinpx0R3IReuwQ2O ewfg9KKMj3vbegzaS4i8CBGlDsPsnNVh62tqlXwJ1w6HvZp+SgTxsAh8GzezNXgB9G4C IcZpAFMPA9hzwj8NGMqHd3KerIM59a2efWSpwoZWQst0uUbNiWKAfLM5CbSatKvifJV+ wwryOnacRVPjRD3QfJCbF5R5QUoLEYwBjo4/hZTMHZpyU2s7KiEbZsIlxH2twAWYOq5M /KsiTOaKGf0HEsGv4WjgrCSCyS7vuoM57MOIOwZWKZ7PMuj8N/eYElJGRaooU0SPvkfe kniQ== X-Gm-Message-State: APjAAAWSc9UEGlel1itkASIdVEP8iS9AZXpQyCHi5tCNZGunSwMDdch1 Zf89Gnl4ZffkERepEe8auCDwSdLPZQ+Ot95Nv4RmKST4 X-Google-Smtp-Source: APXvYqyAWX8IREa9zvjT6fGwqzypX6zbLWJexJZ0g4VwbYWhrP5w7/+Z3lLJJPwYN9Gru0cvikfM+aJq37BAc81i414= X-Received: by 2002:a6b:fe07:: with SMTP id x7mr20793855ioh.3.1554878503321; Tue, 09 Apr 2019 23:41:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ashwin Sai Shankar Date: Tue, 9 Apr 2019 23:41:32 -0700 Message-ID: Subject: Re: Difference between Kubernetes Executor vs PodOperator To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000dc278c0586275c9a" --000000000000dc278c0586275c9a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks, James and Kamil! Please let me know if you have any examples of setting up Kubernetes Executor and Operator. On Tue, Apr 9, 2019 at 8:03 AM James Meickle wrote: > Yes, that summary is correct - the Executor is using Kubernetes to execut= e > all Airflow tasks (each wrapped by a temporary Airflow process), while th= e > PodOperator is using Kubernetes only for that task, to execute one Pod > (which likely won't run any Airflow code at all). > > On Tue, Apr 9, 2019 at 3:17 AM Kamil Ga=C5=82uszka w= rote: > > > Hi Ashwin, > > > > I had exactly same question couple of days ago. Let me try to explain. = If > > I'm wrong please someone correct me. > > > > Kubernetes Executor is used to execute TaskInstance, which means that P= od > > is created of that TaskInstance that for ex. could be BashOperator or > > SlackAPIOperator and after execution of this Task is finished, executor > pod > > is cleaned up/removed. > > > > If you use KubernetesPodOperator, then there could be used whatever > > language/library from docker image that will be scheduled by airflow. F= or > > example You parse CSV files with XSV library in Rust, then > > KubernetesPodOperator is perfect for You as you don't have to spend to > much > > time figuring out how to write custom operator, You just have an image > with > > executable that is simply used with KubernetesPodOperator. > > > > If You use KubernetesPodOperator with KubernetesExecutor then following > > thing will happen: > > > > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which > will > > create KubernetesPodOperator Pod in k8s. > > > > KubernetesPodOperator can be used with any other executors to my > knowledge. > > > > Thanks > > Kamil > > > > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar > > wrote: > > > > > Hi Airflow users, > > > What is the difference between Kube executor vs pod operator? > > > http://airflow.apache.org/kubernetes.html > > > > > > Thanks, > > > Ash > > > > > > --000000000000dc278c0586275c9a--