Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <488819222.50891265319808468.JavaMail.jira@brutus.apache.org>
Date: Thu, 4 Feb 2010 21:43:28 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-326) The lowest level map-reduce APIs
 should be byte oriented
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829801#action_12829801 ] 

Owen O'Malley commented on MAPREDUCE-326:
-----------------------------------------

I agree with Todd. I don't think taking a performance hit in Java by adding memory copies is acceptable.

I think that keeping keys and values separate is useful for a couple of reasons:
  1. keys compress better separately than intermixed with values (not that we do that currently...)
  2. combining the keys and values will effectively block using memcmp for the sort (and it is pretty ugly to consider serializing the "key" fields differently that the "value" fields)
  3. We could support more flexible memory management if the values can be distinguished from the keys.
  4. Since Sequence Files and T Files contain both Keys and Values, they would need to be wrapped together to present them as a single object.

I don't see what this abstraction is buying you over using ByteBuffer and a Serializer that knows how to use it. You could define your Mapper and/or Reducer any way you want and deserialization would be completely in the application's hands. Very much like Google's MapReduce.

> The lowest level map-reduce APIs should be byte oriented
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-326
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-326
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: eric baldeschwieler
>
> As discussed here:
> https://issues.apache.org/jira/browse/HADOOP-1986#action_12551237
> The templates, serializers and other complexities that allow map-reduce to use arbitrary types complicate the design and lead to lots of object creates and other overhead that a byte oriented design would not suffer.  I believe the lowest level implementation of hadoop map-reduce should have byte string oriented APIs (for keys and values).  This API would be more performant, simpler and more easily cross language.
> The existing API could be maintained as a thin layer on top of the leaner API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.