hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akira Ajisaka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6795) Update the document for JobConf#setNumReduceTasks
Date Mon, 31 Oct 2016 03:14:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15621073#comment-15621073

Akira Ajisaka commented on MAPREDUCE-6795:

I'm thinking it's more difficult to tune the value in MRv2 than MRv1.

bq. I think the value numNodes * mapreduce.tasktracker.reduce.tasks.maximum in MRV1 is equal
to the current config mapreduce.job.running.reduce.limit.
mapreduce.job.running.reduce.limit is an optional parameter (MAPREDUCE-5583). The number of
reduce tasks is limited to "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb",
but the resource is shared by map tasks and other applications. Therefore the limit of the
number of reduce tasks becomes smaller than "yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb"
if some map tasks or other applications are running.

After all, "multiplied by (<available memory for reduce tasks> / mapreduce.reduce.memory.mb)"
is good to me.

> Update the document for JobConf#setNumReduceTasks
> -------------------------------------------------
>                 Key: MAPREDUCE-6795
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6795
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Akira Ajisaka
>            Assignee: Yiqun Lin
> The following document is for MRv1. We should update the document for MapReduce on YARN.
> {code:title=JobConf.java}
>    * <b id="NoOfReduces">How many reduces?</b>
>    * 
>    * <p>The right number of reduces seems to be <code>0.95</code> or

>    * <code>1.75</code> multiplied by (&lt;<i>no. of nodes</i>&gt;
>    * <a href="{@docRoot}/../mapred-default.html#mapreduce.tasktracker.reduce.tasks.maximum">
>    * mapreduce.tasktracker.reduce.tasks.maximum</a>).
>    * </p>
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message