airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Meickle <jmeic...@quantopian.com.INVALID>
Subject Re: Difference between Kubernetes Executor vs PodOperator
Date Tue, 09 Apr 2019 15:02:58 GMT
Yes, that summary is correct - the Executor is using Kubernetes to execute
all Airflow tasks (each wrapped by a temporary Airflow process), while the
PodOperator is using Kubernetes only for that task, to execute one Pod
(which likely won't run any Airflow code at all).

On Tue, Apr 9, 2019 at 3:17 AM Kamil GaƂuszka <kamil@flyrlabs.com> wrote:

> Hi Ashwin,
>
> I had exactly same question couple of days ago. Let me try to explain. If
> I'm wrong please someone correct me.
>
> Kubernetes Executor is used to execute TaskInstance, which means that Pod
> is created of that TaskInstance that for ex. could be BashOperator or
> SlackAPIOperator and after execution of this Task is finished, executor pod
> is cleaned up/removed.
>
> If you use KubernetesPodOperator, then there could be used whatever
> language/library from docker image that will be scheduled by airflow. For
> example You parse CSV files with XSV library in Rust, then
> KubernetesPodOperator is perfect for You as you don't have to spend to much
> time figuring out how to write custom operator, You just have an image with
> executable that is simply used with KubernetesPodOperator.
>
> If You use KubernetesPodOperator with KubernetesExecutor then following
> thing will happen:
>
> AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which will
> create KubernetesPodOperator Pod in k8s.
>
> KubernetesPodOperator can be used with any other executors to my knowledge.
>
> Thanks
> Kamil
>
> On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> <ashankar@slack-corp.com.invalid> wrote:
>
> > Hi Airflow users,
> > What is the difference between Kube executor vs pod operator?
> > http://airflow.apache.org/kubernetes.html
> >
> > Thanks,
> > Ash
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message