airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergio B (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-1314) Airflow kubernetes integration
Date Thu, 03 May 2018 08:34:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462104#comment-16462104
] 

Sergio B edited comment on AIRFLOW-1314 at 5/3/18 8:33 AM:
-----------------------------------------------------------

[~dimberman] [~Fokko]

I was wondering if we should share parameters in the [kubernetes] section of the airflow
config file across kubernetes executor and operator. I am not very familiar with airflow components
so I want to hear your opinion before write some code.

I think it would be great for example share namespace configuration, I think that by default
you can share the namespace configuration and if you need you can always set a different namespace
in the dag config. Also for example if you define a default resource policy, labels or annotations
for all your airflow pods you can define in the airflow config file.
 * make it sense read this configuration parameters in the kubernetes operator and share
with executor?
 * would be better create another section in the config file?
 * is there other place where tuning your default operator values?

 


was (Author: ese):
[~dimberman] [~Fokko]

I was wondering if we should share parameters in the [kubernetes] section of the airflow
config file across kubernetes executor and operator. I am not very familiar with airflow components
so I want to hear your opinion before write some code.

I think it would be great for example share namespace configuration, I think that by default
you can share the namespace configuration and if you need you can always set a different namespace
in the dag config. Also for example if you define a default resource policy, labels or annotations
for all your airflow pods you can define in the airflow config file.
 * make it sense read this configuration parameters for the kubernetes operator and share
with executor?
 * would be better create another section in the config file?
 * is there other place where tuning your default operator values?

 

> Airflow kubernetes integration
> ------------------------------
>
>                 Key: AIRFLOW-1314
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1314
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.0
>            Reporter: Daniel Imberman
>            Assignee: Daniel Imberman
>            Priority: Major
>              Labels: features
>             Fix For: 1.10.0
>
>
> Kubernetes is a container-based cluster management system designed by google for easy
application deployment. Companies such as Airbnb, Bloomberg, Palantir, and Google use kubernetes
for a variety of large-scale solutions including data science, ETL, and app deployment. Integrating
airflow into Kubernetes would increase viable use cases for airflow, promote airflow as a
de facto workflow scheduler for Kubernetes, and create possibilities for improved security
and robustness within airflow. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message