airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Edmiston <tedmis...@gmail.com>
Subject Re: Regarding airflow 1.10
Date Wed, 22 Aug 2018 19:09:43 GMT
Hemanth -

To add to what Fokko mentioned...

You can find the KubernetesExecutor implementation in KubernetesExecutor
<https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L465>
and AirflowKubernetesScheduler
<https://github.com/apache/incubator-airflow/blob/7ea7dd2031c32a5dd14e4fd74f415f938a69e3ad/airflow/contrib/executors/kubernetes_executor.py#L292>.
There's also a usage example
<https://github.com/apache/incubator-airflow/blob/272952a9dce932cb2c648f82c9f9f2cafd572ff1/airflow/example_dags/example_kubernetes_executor.py>
.

And here's the KubernetesPodOperator
<https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/contrib/operators/kubernetes_pod_operator.py#L33>
implementation
which has an example
<https://github.com/apache/incubator-airflow/blob/9d516c7134eb22a3c2fc63cf96626ef6e8b247f2/airflow/example_dags/example_kubernetes_operator.py>
as
well.

There are brief docs
<https://airflow.readthedocs.io/en/latest/kubernetes.html> available for
both as well.

And a shameless plug — we've been building an open source Airflow on
Kubernetes stack at Astronomer (Helm charts
<https://github.com/astronomerio/helm.astronomer.io>, Docker images
<https://github.com/astronomerio/astronomer>).  Some people use this to
avoid having to rebuild all of the containerization.  Currently we use
CeleryExecutor but are already migrating toward the KubernetesExecutor.
Our goal is to provide the best experience for running Airflow on
Kubernetes.  Feel free to message me off list if this is interesting.

Best,
Taylor

*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>



On Tue, Aug 21, 2018 at 10:15 AM Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi Hemanth,
>
> Thanks for the question. There is the KubernetesExecutor which just uses a
> container with Airflow to schedule a task on the kubernetes cluster. This
> container will then execute a single task, and then the Executor checks the
> exitcode of the container.
>
> There is also the KubernetesOperator which allows you to kick of a new
> container on the kubernetes cluster. Hope this helps.
>
> Cheers, Fokko
>
> Op di 21 aug. 2018 om 15:54 schreef Musunuru, Hemanth <
> Hemanth.Musunuru@nike.com>
>
> > Hi Team,
> >
> > Does Kubernetes executor in 1.10 create new  pod for every task ? (Which
> > means won’t there be any active workers running except for new Tasks)
> >
> > We are trying to containerize our existing to airflow environment, your
> > reply Will help us to understand it more.
> >
> > Thanks in Advance .
> >
> > Thanks
> > Hemanth
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message