airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-398) Option to use cgroups to limit task resource utilization
Date Fri, 20 Jan 2017 15:44:26 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831976#comment-15831976
] 

ASF subversion and git services commented on AIRFLOW-398:
---------------------------------------------------------

Commit b56cb5cc97de074bb0e520f66b79e7eb2d913fb1 in incubator-airflow's branch refs/heads/v1-8-test
from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b56cb5c ]

[AIRFLOW-219][AIRFLOW-398] Cgroups + impersonation

Submitting on behalf of plypaul

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-219
-
https://issues.apache.org/jira/browse/AIRFLOW-398

Testing Done:
- Running on Airbnb prod (though on a different
mergebase) for many months

Credits:
Impersonation Work: georgeke did most of the work
but plypaul did quite a bit of work too.
Cgroups: plypaul did most of the work, I just did
some touch up/bug fixes (see commit history,
cgroups + impersonation commit is actually plypaul
's not mine)

Closes #1934 from aoen/ddavydov/cgroups_and_impers
onation_after_rebase


> Option to use cgroups to limit task resource utilization
> --------------------------------------------------------
>
>                 Key: AIRFLOW-398
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-398
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: core
>            Reporter: Paul Yang
>            Assignee: Paul Yang
>            Priority: Minor
>
> Now that tasks have resource requirements specified, Airflow can use the defined resource
requirements as limits when running the Airflow tasks. A common use case is to limit the memory
used by a task so that it doesn't cause the worker machine to go OOM.
> The proposed idea is to implement a TaskRunner class that, by default, executes the Airflow
run command in a bash subprocess as it is currently done now.
> Alternatively, the user can specify a TaskRunner class that uses cgroups to ensure that
resource requirements are respected. This would be a lightweight approach to limiting resource
usage by tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message