hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj K (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
Date Fri, 14 Oct 2016 04:32:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574159#comment-15574159

Devaraj K commented on MAPREDUCE-6749:

Thanks [~Naganarasimha] for the responses and discussion.
bq. hope we can create a new branch and get the things in there so that its easier for others
to have a look before it gets into the trunk or main stream branches
Sure, I will create a branch for this.

bq. If there was a way for admin to enforce it it would be usefull. If its just client level
configuration it just adds into already long list of configurations and users will not be
clear what to configure for it. And besides would it be better to have just how many tasks
can reuse a given container and try to avoid for Map and Reduce seperately ?
I am thinking that we can provide comprehensive way to control this feature, I can convince
that it will be an another configuration for the user to configure it. We can discuss this

bq. Btw it could be also good to introduce a metric for number of Map Tasks or Reduce tasks
which has reused the containers
Good thought, we can have metrics for this.

bq. This was the problem which we generally faced and difficult for the customers to understand
that entire log is not for the task attempt, so was wondering to have any better approach
to this.
We can think of displaying the part of the container log(logs generated for a task attempt)
in JHS Web UI instead of the whole container logs by discarding the other task attempt logs.
Do you think this sounds ok or any better way?

> MR AM should reuse containers for Map/Reduce Tasks
> --------------------------------------------------
>                 Key: MAPREDUCE-6749
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers for Map/Reduce
Tasks similar to the JVM Reuse feature we had in MRv1.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message