hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj K (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
Date Thu, 06 Oct 2016 04:53:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550874#comment-15550874

Devaraj K commented on MAPREDUCE-6749:

Thanks [~Naganarasimha] for looking into this and looking forward to work on this with you.

Is it required to have number of total Maps(/reducers) to be run using container reuse? i
felt we can just keep whether to reuse the containers are not and one assumption that map
containers will be reused only for map and reduce only for reduce
I think the limit configuration for no of map/reduce reuse containers would allow other applications
to start running without waiting for the Job to be finished when reuse is enabled. If there
is a big Job running which could occupy the entire cluster, and then any high priority application
gets submitted this limit for maps/reduce container would probably give a room for high priority
application to start running without preempting the containers of the previous Job. By default
there is no limit for number of containers to be reused and if any user/Job wanted to have
this constraint they can configure it. 

did not understand the need of RMContainerReuseRequestor.java, may be i will go through MAPREDUCE-6773
but was expecting more like just TaskAttemptIimpl gets TA_DONE event was expecting it do have
a new ReuseContainerTransition where in if required it will reassign a new task to it else
will continue to CLEANUP_CONTAINER_TRANSITION. may be we can discuss more on this in MAPREDUCE-6773
RMContainerReuseRequestor has the functionality for checking the container reuse constraints
and adding to reuse containers if it is eligible. This class abstracts the reuse functionality
from regular RMContainerRequestor and RMContainerReuseRequestor will be used only when the
RMContainer reuse is enabled. If you want to have a try this feature, you can apply MAPREDUCE-6773,
MAPREDUCE-6781, MAPREDUCE-6784, MAPREDUCE-6785, MAPREDUCE-6786 and then try this feature.
I would be happy to get your feedback or suggestions if any.

If we have some conclusion on issue 2, then we can think of whether only TaskAttemptImpl state
is sufficient or we need to modify others too.
Sure, We can discuss it further if required.

{quote} Logs generated by the tasks in a container will be appended in the same container
log file we need to ensure we are able to display trhough JHS and other places where ever
required to display
TaskAttempt is linked to the container and the corresponding container log would be displayed
for TaskAttempt through the JHS. Here we should note that the whole container log which is
displaying for TaskAttempt is not applicable to the TaskAttempt and the log can be identified
easily which part applicable to it. 

> MR AM should reuse containers for Map/Reduce Tasks
> --------------------------------------------------
>                 Key: MAPREDUCE-6749
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers for Map/Reduce
Tasks similar to the JVM Reuse feature we had in MRv1.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message