airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwin Sai Shankar <ashan...@slack-corp.com.INVALID>
Subject Re: Difference between Kubernetes Executor vs PodOperator
Date Wed, 10 Apr 2019 06:41:32 GMT
Thanks, James and Kamil! Please let me know if you have any examples of
setting up Kubernetes Executor and Operator.

On Tue, Apr 9, 2019 at 8:03 AM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> Yes, that summary is correct - the Executor is using Kubernetes to execute
> all Airflow tasks (each wrapped by a temporary Airflow process), while the
> PodOperator is using Kubernetes only for that task, to execute one Pod
> (which likely won't run any Airflow code at all).
>
> On Tue, Apr 9, 2019 at 3:17 AM Kamil GaƂuszka <kamil@flyrlabs.com> wrote:
>
> > Hi Ashwin,
> >
> > I had exactly same question couple of days ago. Let me try to explain. If
> > I'm wrong please someone correct me.
> >
> > Kubernetes Executor is used to execute TaskInstance, which means that Pod
> > is created of that TaskInstance that for ex. could be BashOperator or
> > SlackAPIOperator and after execution of this Task is finished, executor
> pod
> > is cleaned up/removed.
> >
> > If you use KubernetesPodOperator, then there could be used whatever
> > language/library from docker image that will be scheduled by airflow. For
> > example You parse CSV files with XSV library in Rust, then
> > KubernetesPodOperator is perfect for You as you don't have to spend to
> much
> > time figuring out how to write custom operator, You just have an image
> with
> > executable that is simply used with KubernetesPodOperator.
> >
> > If You use KubernetesPodOperator with KubernetesExecutor then following
> > thing will happen:
> >
> > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which
> will
> > create KubernetesPodOperator Pod in k8s.
> >
> > KubernetesPodOperator can be used with any other executors to my
> knowledge.
> >
> > Thanks
> > Kamil
> >
> > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> > <ashankar@slack-corp.com.invalid> wrote:
> >
> > > Hi Airflow users,
> > > What is the difference between Kube executor vs pod operator?
> > > http://airflow.apache.org/kubernetes.html
> > >
> > > Thanks,
> > > Ash
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message