hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
Date Sun, 23 Oct 2016 18:42:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15600121#comment-15600121

Naganarasimha G R commented on MAPREDUCE-6773:

hi @devaraj k 
Sorry from my side as i took a while to reply back on this jira. I think we need to cancel
the patch and rebase it based on the work done in MAPREDUCE-6772. 
And coming to some major discussion points :
# Generally i like the approach of modifying the existing hierarchy i.e {{RMCommunicator <-
RMContainerRequestor <- RMContainerAllocator}} to {{RMCommunicator <- RMContainerAllocator}}
and having a new hierarchy {{ContainerReqestor <- RMContainerRequestor <- RMContainerReUseRequestor}}
which specifically deals with the handling of structures required for the requesting, but
any other better name as it not exactly requesting the container but just storing the stats
required for it ?
# IIUC, Though superficially {{container.reuse.max-maptaskcontainers}} looks usefull, again
was wondering how the job submitter/admin is going to configure this max # of concurrent MAP(/reduce)
container which can be reused ? atleast the previous one(container.reuse.max-maptasks) based
on the historic values, admin can determine how much time one map/reduce task is going to
take time and accordingly ask the job submitter to configure but this concurrent max reuse
(MAP/REDUCE) tasks is just a blind configuration and would be difficult to determine the right
value. Thoughts ? (hope my understanding of this config is correct !).
# may be we can split this jira into 2 parts like creating the interface for ContainerReqestor
and either in new jira or in MAPREDUCE-6786, add the code related to reuse so that its complete.
as these jiras doesnt look to be testable pieces of jiras.

nits :
# has white space errors when applying the patch
# if we plan to use MR_AM_CONTAINER_REUSE_MAX_MAPTASKCONTAINERS (also reduce one), then we
need to rename the prefix from {{yarn.app.mapreduce.am}} to {{yarn.app.mapreduce}} & *mapred-default.xml*
needs to be updated
# {{JOB_CONTAINER_COMPLETED}}, is it related to this patch ? should it be in MAPREDUCE-6784?
seems like not used anywhere in the current patch.
# {{ContainerRequestorEvent}} has almost similar naming to that of {{ContainerRequestEvent}},
further was wondering whether we can rename it to ContainerReUseRequestEvent, as its only
used by RMContainerReUseRequestor.

> Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
> -------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-6773
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: applicationmaster, mrv2
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch
> Add RM Container Reuse Requestor which handles the reuse containers against the Job reource

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message