hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
Date Thu, 07 Mar 2013 04:52:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595550#comment-13595550
] 

Mariappan Asokan commented on MAPREDUCE-3685:
---------------------------------------------

Hi Ravi,
  I guess I am too late to comment since your patch has been committed already.  In any case,
I have the following comments since you asked:)

* In {{closeOnDiskFile()}} the following lines of code
{code}
    if (onDiskMapOutputs.size() >= (2 * ioSortFactor - 1)) {
      onDiskMerger.startMerge(onDiskMapOutputs);
    }
{code}
can be changed to
{code}
    if (onDiskMapOutputs.size() >= ioSortFactor) {
      onDiskMerger.startMerge(onDiskMapOutputs);
    }
{code}
Please confirm.
* In the class {{CompressAwarePath}} there is a nit in {{compareTo().}}  The following lines:
{code}
        } else if (this.getCompressedSize() > compPath.getCompressedSize()) {
          return 1;
{code}
can be simplified as:
{code}
        } else {
          return 1;
{code}
The set will be partially ordered without an additional compare and without executing the
line
{code}
      return super.compareTo(obj);
{code}
* Since the patch fixes some performance issues, did you have a chance to run some benchmarks
that show improvements?  I know this will take some time.  I will leave it to you.

-- Asokan

                
> There are some bugs in implementation of MergeManager
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-3685
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.1
>            Reporter: anty.rao
>            Assignee: anty
>            Priority: Critical
>             Fix For: 0.23.7, 2.0.4-beta
>
>         Attachments: MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch,
MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch,
MAPREDUCE-3685.patch, MAPREDUCE-3685.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message