hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-326) The lowest level map-reduce APIs should be byte oriented
Date Thu, 04 Feb 2010 21:43:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829801#action_12829801

Owen O'Malley commented on MAPREDUCE-326:

I agree with Todd. I don't think taking a performance hit in Java by adding memory copies
is acceptable.

I think that keeping keys and values separate is useful for a couple of reasons:
  1. keys compress better separately than intermixed with values (not that we do that currently...)
  2. combining the keys and values will effectively block using memcmp for the sort (and it
is pretty ugly to consider serializing the "key" fields differently that the "value" fields)
  3. We could support more flexible memory management if the values can be distinguished from
the keys.
  4. Since Sequence Files and T Files contain both Keys and Values, they would need to be
wrapped together to present them as a single object.

I don't see what this abstraction is buying you over using ByteBuffer and a Serializer that
knows how to use it. You could define your Mapper and/or Reducer any way you want and deserialization
would be completely in the application's hands. Very much like Google's MapReduce.

> The lowest level map-reduce APIs should be byte oriented
> --------------------------------------------------------
>                 Key: MAPREDUCE-326
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-326
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: eric baldeschwieler
> As discussed here:
> https://issues.apache.org/jira/browse/HADOOP-1986#action_12551237
> The templates, serializers and other complexities that allow map-reduce to use arbitrary
types complicate the design and lead to lots of object creates and other overhead that a byte
oriented design would not suffer.  I believe the lowest level implementation of hadoop map-reduce
should have byte string oriented APIs (for keys and values).  This API would be more performant,
simpler and more easily cross language.
> The existing API could be maintained as a thin layer on top of the leaner API.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message