hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Snyder <bruce.sny...@gmail.com>
Subject Re: Introducing Cloud MapReduce
Date Fri, 27 Nov 2009 21:18:35 GMT
On Fri, Nov 27, 2009 at 11:34 AM, Tim Robertson
<timrobertson100@gmail.com> wrote:
> Hi Bruce,
> Interesting stuff.  It looks like it only works with String Keys and
> Values (possibly a reason you see large performance gains with simpler
> serialization requirements) - have you any plans to support other
> types in the roadmap? Perhaps with Protobufs or Avro serialization?

I can definitely see the need for supporting more than one type of
key/value. We'll need to add this to the roadmap, thanks.

> Have you considered using the same Mapper and Reducer class and method
> signatures so that a MR job could be written and launched against
> CloudMR or Hadoop?
> [The reason I ask is I am hacking an implementation of a single JVM,
> multithreaded MR framework that does this, so I can ship the same
> analysis I run on large datasets to people to run on small datasets
> also (much lighter than Hadoop but compatible).  I am putting it on
> google code mapreduce4j.  It might be nice if there was a standard MR
> API that people coded against and launched in various frameworks -
> just a thought].

Excellent points, Tim. A point that I raised was trying to determine a
standard sort of API that would work across implementations. So far
I'm not sure if this means that we match what is already out there or
if we provide interfaces for many languages or what. I like the idea,
but I'll let Huan chime in here.

perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

View raw message