Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Wed, 10 Jun 2015 10:51:04 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12823146.1429731218000.36423.1433933464759@Atlassian.JIRA>
In-Reply-To: <JIRA.12823146.1429731218000@Atlassian.JIRA>
References: <JIRA.12823146.1429731218000@Atlassian.JIRA>
 <JIRA.12823146.1429731218659@arcas>
Subject: [jira] [Commented] (MAPREDUCE-6332) Provide facility to users for
 writting custom MergeManager implementation when custom
 shuffleconsumerPluggin is used
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580385#comment-14580385 ] 

Devaraj K commented on MAPREDUCE-6332:
--------------------------------------

Thanks [~rohithsharma] for the patch. Here are some of my comments,

1. InMemoryReader still uses MergeManagerImpl, can you update the references to MergeManager.
2. I don't think it is a good idea to have static inner class in the exposing interface. Can we move CompressAwarePath as a separate class instead of having it in MergeManager?
3. 
{code}
void closeInMemoryFile(InMemoryMapOutput<K, V> mapOutput);
{code}
Here we are providing API with  parameter type as InMemoryMapOutput, but InMemoryMapOutput marked as private class and also the visibility is default which cannot be accessed outside of the package.
4. {code:xml}
void closeOnDiskFile(CompressAwarePath compressAwarePath);
{code}
CompressAwarePath is used mostly inside the class MergeManagerImpl except one place in OnDiskMapOutput. Can we avoid giving the argument type as CompressAwarePath here?
5. Can we have detailed java doc for MergeManager at class level, how to implement it for custom merge manager? And also please add detailed description for the newly added API's in MergeManager. 
6. Can we have a test to demonstrate how do we configure and use the custom merge manager?
7. And also there are tests using MergeManagerImpl type for references, please update them as well.


> Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6332
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Rohith
>            Assignee: Rohith
>         Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch
>
>
> MR provides ability to the user for plugin custom ShuffleConsumerPlugin using *mapreduce.job.reduce.shuffle.consumer.plugin.class*.  When the user is allowed to use this configuration as plugin, user also interest in implementing his own MergeManagerImpl. 
> But now , user is forced to use MR provided MergeManagerImpl instead of custom MergeManagerImpl when user is using shuffle.consumer.plugin class. There should be well defined API's in MergeManager that can be used for any implementation without much effort to user for custom implementation.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)