airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Imberman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2955) Kubernetes pod operator: Unable to set requests/limits on task pods
Date Wed, 29 Aug 2018 23:37:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596934#comment-16596934
] 

Daniel Imberman commented on AIRFLOW-2955:
------------------------------------------

[~jpds] So the problem here is that the operator is expecting a "Resources" class [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/pod.py#L19.]
It might actually make more sense to have users just use a dict and then generate that class
ourselves. Should be a pretty easy fix. Until then try creating that class and it should work.

> Kubernetes pod operator: Unable to set requests/limits on task pods
> -------------------------------------------------------------------
>
>                 Key: AIRFLOW-2955
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2955
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Jon Davies
>            Priority: Major
>
> When I try and set a resource limit/request on a DAG task with the KubernetesPodOperator
as follows:
> {code:java}
> resources={"limit_cpu": 1, "request_cpu": 1},
> {code}
> ...I get:
> {code:java}
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task Traceback
(most recent call last):
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/bin/airflow", line 32, in <module>
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task     args.func(args)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task     return
f(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 498, in run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task     _run(args,
dag, ti)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 402, in _run
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task     pool=args.pool,
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task     return
func(*args, **kwargs)
> [2018-08-24 15:51:27,795] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/models.py", line 1633, in _run_raw_task
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     result
= task_copy.execute(context=context)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
line 115, in execute
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     get_logs=self.get_logs)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line
71, in run_pod
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     resp
= self.run_pod_async(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line
52, in run_pod_async
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     req
= self.kube_req_factory.create(pod)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py",
line 56, in create
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     self.extract_resources(pod,
req)
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task   File
"/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py",
line 160, in extract_resources
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task     if
not pod.resources or pod.resources.is_empty_resource_request():
> [2018-08-24 15:51:27,796] {base_task_runner.py:107} INFO - Job 2: Subtask task AttributeError:
'dict' object has no attribute 'is_empty_resource_request'
> {code}
> ...setting https://github.com/apache/incubator-airflow/blob/fc10f7e0a04145a0b2f31f8d0990bbe900b4e8a2/airflow/example_dags/example_kubernetes_executor.py#L66
works, however that only adjusts the metadata for the worker pod and not the pod ultimately
used for the task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message