airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil GaƂuszka <ka...@flyrlabs.com>
Subject Re: Difference between Kubernetes Executor vs PodOperator
Date Tue, 09 Apr 2019 07:16:08 GMT
Hi Ashwin,

I had exactly same question couple of days ago. Let me try to explain. If
I'm wrong please someone correct me.

Kubernetes Executor is used to execute TaskInstance, which means that Pod
is created of that TaskInstance that for ex. could be BashOperator or
SlackAPIOperator and after execution of this Task is finished, executor pod
is cleaned up/removed.

If you use KubernetesPodOperator, then there could be used whatever
language/library from docker image that will be scheduled by airflow. For
example You parse CSV files with XSV library in Rust, then
KubernetesPodOperator is perfect for You as you don't have to spend to much
time figuring out how to write custom operator, You just have an image with
executable that is simply used with KubernetesPodOperator.

If You use KubernetesPodOperator with KubernetesExecutor then following
thing will happen:

AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which will
create KubernetesPodOperator Pod in k8s.

KubernetesPodOperator can be used with any other executors to my knowledge.

Thanks
Kamil

On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
<ashankar@slack-corp.com.invalid> wrote:

> Hi Airflow users,
> What is the difference between Kube executor vs pod operator?
> http://airflow.apache.org/kubernetes.html
>
> Thanks,
> Ash
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message