hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wiley <kwi...@keithwiley.com>
Subject Re: installing pipes on os x
Date Mon, 21 May 2012 17:57:26 GMT
Hubris aside (mine I mean), I'm probably not qualified to be the last word on this with regard
to advising you on your project's overall direction, so hopefully others will chime in on
this discussion...but that said, when I started with hadoop a few years ago and our project
exceed the capabilities of direct-Java hadoop interfacing, we briefly entertained the notion
of pipes and learned (to the best of my recollection) that pipes was officially losing its
support within the hadoop community (it may have even been depricated although I don't remember
for sure) and that streaming was being officially advocated as the replacement (unless I am
grossly exaggerating the advice I received at the time).  In fact, the primary reason I abandoned
pipes wasn't so much that the technology itself wouldn't work, but that there was far less
community support for it than for streaming, so whenever I got stuck I couldn't really ask
for help (or I could ask but I would be inquiring a much smaller pool of potential responders,
thus decreasing the scope of any response).

At any rate, I have been quite satisfied with streaming.  Gotchas have included coding from
a scratch an entire partially abstract C++ class family to offer a streaming interface to
communicate between hadoop and underlying native processing, to say nothing of coding up binary/ascii
conversion in a variety of formats to shuffle data across the i/o streams (for reasons I can't
recall now, the binary streaming interface wasn't a good option for us at the time).

Still, all's well.

Best of luck.

On May 21, 2012, at 10:26 , Peter Cogan wrote:

> Hi Keith, 
> Yeah - maybe that was my first mistake. Since I haven't had much luck getting pipes working
I've started looking at streaming. I am still 'early in my hadoop efforts' so now is the time
for big course corrections
> thanks
> Peter
> On 21 May 2012, at 18:11, Keith Wiley wrote:
>> Mind if I ask why you prefer pipes to streaming?  I abandoned pipes early in my hadoop
efforts and went all engines full on streaming.
>> On May 21, 2012, at 04:07 , Peter Cogan wrote:
>>> Hi,
>>> I am trying to install pipes on os x following the instructions here

Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com

"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
                                           --  Mark Twain

View raw message