airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Davies (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-2920) Kubernetes pod operator: namespace is a hard requirement
Date Mon, 20 Aug 2018 09:36:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16585694#comment-16585694
] 

Jon Davies edited comment on AIRFLOW-2920 at 8/20/18 9:35 AM:
--------------------------------------------------------------

Workaround appears to be to set

{code:java}
namespace=os.getenv('AIRFLOW_KUBE_NAMESPACE'),
{code}

...in the DAG code and then specify that variable in the scheduler configuration. But then
the executor worker pod that gets spun up fails with:

{noformat}
[2018-08-20 09:32:12,003] {base_task_runner.py:107} INFO - Job 2: Subtask thing-downloader
ValueError: Missing the required parameter `namespace` when calling `create_namespaced_pod
{noformat}



was (Author: jpds):
Workaround appears to be to set

{code:java}
namespace=os.getenv('AIRFLOW_KUBE_NAMESPACE'),
{code}

...in the DAG code and then specify that variable in the scheduler configuration.

> Kubernetes pod operator: namespace is a hard requirement
> --------------------------------------------------------
>
>                 Key: AIRFLOW-2920
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2920
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Jon Davies
>            Priority: Major
>
> Hello,
> I'm using the Kubernetes pod operator for my DAGs, I install Airflow to its own namespace
within my Kubernetes cluster (for example: "testing-airflow") and I would like pods spun up
by that Airflow instance to live in that namespace.
> However, I have to hardcode the namespace into my DAG definition code and so I have to
rebuild the Docker image for Airflow to be able to spin up a "production-airflow" namespace
as the namespace is a hard requirement in the Python code - it'd be nice if the DAG could
just default to its own namespace if none is defined.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message