airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <daniel.imber...@gmail.com>
Subject Re: Regarding airflow 1.10
Date Wed, 22 Aug 2018 23:37:32 GMT
Hi  Hemanth,

Fokko and Taylor have covered this question but just cc'ing myself in case
you have any further questions about the k8s executor :).

On Wed, Aug 22, 2018 at 12:10 PM Taylor Edmiston <tedmiston@gmail.com>
wrote:

> Hemanth -
>
> To add to what Fokko mentioned...
>
> You can find the KubernetesExecutor implementation in KubernetesExecutor
> <
> https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L465
> >
> and AirflowKubernetesScheduler
> <
> https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L292
> >.
> There's also a usage example
> <
> https://github.com/apache/incubator-airflow/blob/272952a9dce932cb2c648f82c9f9f2cafd572ff1/airflow/example_dags/example_kubernetes_executor.py
> >
> .
>
> And here's the KubernetesPodOperator
> <
> https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/contrib/operators/kubernetes_pod_operator.py#L33
> >
> implementation
> which has an example
> <
> https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/example_dags/example_kubernetes_operator.py
> >
> as
> well.
>
> There are brief docs
> <https://airflow.readthedocs.io/en/latest/kubernetes.html> available for
> both as well.
>
> And a shameless plug — we've been building an open source Airflow on
> Kubernetes stack at Astronomer (Helm charts
> <https://github.com/astronomerio/helm.astronomer.io>, Docker images
> <https://github.com/astronomerio/astronomer>).  Some people use this to
> avoid having to rebuild all of the containerization.  Currently we use
> CeleryExecutor but are already migrating toward the KubernetesExecutor.
> Our goal is to provide the best experience for running Airflow on
> Kubernetes.  Feel free to message me off list if this is interesting.
>
> Best,
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
>
> On Tue, Aug 21, 2018 at 10:15 AM Driesprong, Fokko <fokko@driesprong.frl>
> wrote:
>
> > Hi Hemanth,
> >
> > Thanks for the question. There is the KubernetesExecutor which just uses
> a
> > container with Airflow to schedule a task on the kubernetes cluster. This
> > container will then execute a single task, and then the Executor checks
> the
> > exitcode of the container.
> >
> > There is also the KubernetesOperator which allows you to kick of a new
> > container on the kubernetes cluster. Hope this helps.
> >
> > Cheers, Fokko
> >
> > Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth <
> > Hemanth.Musunuru@nike.com>
> >
> > > Hi Team,
> > >
> > > Does Kubernetes executor in 1.10 create new  pod for every task ?
> (Which
> > > means won’t there be any active workers running except for new Tasks)
> > >
> > > We are trying to containerize our existing to airflow environment, your
> > > reply Will help us to understand it more.
> > >
> > > Thanks in Advance .
> > >
> > > Thanks
> > > Hemanth
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message