airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Meickle <jmeic...@quantopian.com.INVALID>
Subject Re: Difference between Kubernetes Executor vs PodOperator
Date Wed, 10 Apr 2019 13:13:28 GMT
We use a fork of the KubernetesPodOperator in house (and we don't use the
Executor). The steps to use it are roughly:

1) Set up your Kubernetes cluster (can be EKS, GKE, etc.)
2) Make sure your Airflow cluster has network access to the Kubernetes API
servers
3) Create a ServiceAccount for the Airflow user, and if using RBAC, grant
it access to whatever you'll be creating with it (for instance, we run all
Airflow pods within an Airflow-specific namespace)
4) Create a ~/.kube/config file to hold your Kubernetes credentials and
define which clusters are available:
https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#define-clusters-users-and-contexts
5) Create a DAG that imports the operator and provides necessary arguments.
It will use the kubectl config file's credentials when submitting to the
cluster.

You may also need a pipeline for building the images that the operator uses.

On Wed, Apr 10, 2019 at 2:41 AM Ashwin Sai Shankar
<ashankar@slack-corp.com.invalid> wrote:

> Thanks, James and Kamil! Please let me know if you have any examples of
> setting up Kubernetes Executor and Operator.
>
> On Tue, Apr 9, 2019 at 8:03 AM James Meickle
> <jmeickle@quantopian.com.invalid> wrote:
>
> > Yes, that summary is correct - the Executor is using Kubernetes to
> execute
> > all Airflow tasks (each wrapped by a temporary Airflow process), while
> the
> > PodOperator is using Kubernetes only for that task, to execute one Pod
> > (which likely won't run any Airflow code at all).
> >
> > On Tue, Apr 9, 2019 at 3:17 AM Kamil GaƂuszka <kamil@flyrlabs.com>
> wrote:
> >
> > > Hi Ashwin,
> > >
> > > I had exactly same question couple of days ago. Let me try to explain.
> If
> > > I'm wrong please someone correct me.
> > >
> > > Kubernetes Executor is used to execute TaskInstance, which means that
> Pod
> > > is created of that TaskInstance that for ex. could be BashOperator or
> > > SlackAPIOperator and after execution of this Task is finished, executor
> > pod
> > > is cleaned up/removed.
> > >
> > > If you use KubernetesPodOperator, then there could be used whatever
> > > language/library from docker image that will be scheduled by airflow.
> For
> > > example You parse CSV files with XSV library in Rust, then
> > > KubernetesPodOperator is perfect for You as you don't have to spend to
> > much
> > > time figuring out how to write custom operator, You just have an image
> > with
> > > executable that is simply used with KubernetesPodOperator.
> > >
> > > If You use KubernetesPodOperator with KubernetesExecutor then following
> > > thing will happen:
> > >
> > > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which
> > will
> > > create KubernetesPodOperator Pod in k8s.
> > >
> > > KubernetesPodOperator can be used with any other executors to my
> > knowledge.
> > >
> > > Thanks
> > > Kamil
> > >
> > > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar
> > > <ashankar@slack-corp.com.invalid> wrote:
> > >
> > > > Hi Airflow users,
> > > > What is the difference between Kube executor vs pod operator?
> > > > http://airflow.apache.org/kubernetes.html
> > > >
> > > > Thanks,
> > > > Ash
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message