hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Feinberg" <a...@socialmedia.com>
Subject Re: Is pipes really supposed to be a serious API? Is it being actively developed?
Date Fri, 19 Sep 2008 18:22:51 GMT
Hi Marc,

I have myself personally used pipes, but I found it's performance wasn't faster
than Java (on the same issue). I also found a lack of documentation (e.g. I saw
nothing about xdr).

What I'd like to see is an example where you're using pipes to pass a struct
from a mapper to a reducer using xdr (I have used bit arrays/packed structures
to pass data from a mapper to a reducer using native byte order since it's safe
to assume that machines in a cluster would be heterogenous).

Perhaps if pipes saw more exposure it'd be used more frequently.


On Fri, Sep 19, 2008 at 11:10 AM, Marc Vaillant <vaillant@fastmail.fm> wrote:
> Only about 5 pipes/c++ related posts since mid July, and basically no
> responses.  Is anyone really using or actively developing pipes?  We've
> invested some time to make it platform independent (ported bsd sockets
> to boost sockets, and the xdr serialization to boost serialization), but
> it's still lacking an efficient way to submit jobs, more efficient
> mechanism for executing map/reduce without launching pipes executables,
> etc.
> Hello, anyone out there?
> Marc

Alex Feinberg
Platform Engineer, SocialMedia Networks

View raw message