hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4900) Dynamic configuration for task slots on TT
Date Wed, 09 Oct 2013 22:14:42 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790919#comment-13790919
] 

Luke Lu commented on MAPREDUCE-4900:
------------------------------------

Thanks for the patch Binglin! Adding a new admin protocol is good idea vs. modifying {{AdminOperationsProtocol}},
which would break backward compatibility.

Reflecting comments on YARN-291 related JIRAs:
# {{DynamicResourceProtocol}}#setNodeSlot set the number of slots one node and one type at
a time, which is inefficient for batch operations. Suggest a more flexible API: {{setDynamicSlots(String
requestJson)}}, where the request json is a list of resource info objects, which could be
easily to extended later. e.g. {code}[{"node": "tt1", "mapslots": 8}, {"node": "tt2"....}]{code}
# Save the original (statically configured) slots info, such that when an admin manually changes
slots in the mapred-site.xml and bounce a TT, the value (if different than the original) can
overrides the dynamic slots. This behavior would be more backward compatible and less surprising
as the dynamic resource protocol is new and mostly used by resource pool schedulers.

> Dynamic configuration for task slots on TT
> ------------------------------------------
>
>                 Key: MAPREDUCE-4900
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4900
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: resourcemanager, tasktracker
>    Affects Versions: 1.1.1
>            Reporter: Junping Du
>            Assignee: Binglin Chang
>         Attachments: MAPREDUCE-4900-demo.patch, MAPREDUCE-4900-demo-with-JMX.patch, MAPREDUCE-4900-demo-with-JMX.v2.patch,
MAPREDUCE-4944-4900.jmx.v3.patch
>
>
> The current Hadoop MRV1 resource management logic assumes per node slot number is static
during the lifetime of the TT process. Allowing run-time configuration on per node slot will
give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
> For more background or design details of this effort, please refer proposal in HADOOP-9165.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message