hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-434) local map-reduce job limited to single reducer
Date Tue, 02 Feb 2010 00:02:22 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron Kimball updated MAPREDUCE-434:

    Attachment: MAPREDUCE-434.2.patch

Submitting new patch sync'd with trunk.

When testing this patch, I uncovered a problematic interaction between mapper/reducer tasks
and the {{Merger}}. Specifically, {{MergeQueue}} uses a static field {{Merger.includeFinalMerge}}
which is set to true in {{MapTask}}, but is never cleared.

Now that both task types can occur in the same JVM, and a {{Merger}} may be used in both (local
reduce tasks did not actually instantiate a {{Merger}} in the old codepath), this flag must
be made a per-{{MergeQueue}} setting. I removed the static field and pushed it into {{MergeQueue}}
as a member field; the {{Merger.merge()}} methods used as constructors for {{MergeQueue}}
now take a {{TaskType}} parameter which can be set to {{TaskType.MAP}} to achieve the same

> local map-reduce job limited to single reducer
> ----------------------------------------------
>                 Key: MAPREDUCE-434
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-434
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>         Environment: local job tracker
>            Reporter: Yoram Arnon
>            Assignee: Aaron Kimball
>            Priority: Minor
>         Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.patch
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and
the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to partition
based on the number of reduce tasks.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message