hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-434) local map-reduce job limited to single reducer
Date Tue, 02 Feb 2010 00:02:22 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Kimball updated MAPREDUCE-434:
------------------------------------

    Attachment: MAPREDUCE-434.2.patch

Submitting new patch sync'd with trunk.

When testing this patch, I uncovered a problematic interaction between mapper/reducer tasks
and the {{Merger}}. Specifically, {{MergeQueue}} uses a static field {{Merger.includeFinalMerge}}
which is set to true in {{MapTask}}, but is never cleared.

Now that both task types can occur in the same JVM, and a {{Merger}} may be used in both (local
reduce tasks did not actually instantiate a {{Merger}} in the old codepath), this flag must
be made a per-{{MergeQueue}} setting. I removed the static field and pushed it into {{MergeQueue}}
as a member field; the {{Merger.merge()}} methods used as constructors for {{MergeQueue}}
now take a {{TaskType}} parameter which can be set to {{TaskType.MAP}} to achieve the same
effect.

> local map-reduce job limited to single reducer
> ----------------------------------------------
>
>                 Key: MAPREDUCE-434
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-434
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>         Environment: local job tracker
>            Reporter: Yoram Arnon
>            Assignee: Aaron Kimball
>            Priority: Minor
>         Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.patch
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and
the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to partition
based on the number of reduce tasks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message