hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3482) Report NM available resources in heartbeat
Date Fri, 17 Apr 2015 16:08:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500097#comment-14500097
] 

Karthik Kambatla commented on YARN-3482:
----------------------------------------

bq. With this and the containers utilization, we can estimate the utilization of external
processes.

True, but I fear that will be too conservative. If we go that route, HBase RegionServers could
grow aggressively and adversely affect resources under Yarn. By having an interface for available
resources, we ensure Yarn aggressively schedules work to claim all available resources. Changing
these available resources could be through a secure interface admins or a white-list of processes
can access. 

> Report NM available resources in heartbeat
> ------------------------------------------
>
>                 Key: YARN-3482
>                 URL: https://issues.apache.org/jira/browse/YARN-3482
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Inigo Goiri
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> NMs are usually collocated with other processes like HDFS, Impala or HBase. To manage
this scenario correctly, YARN should be aware of the actual available resources. The proposal
is to have an interface to dynamically change the available resources and report this to the
RM in every heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message