hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Cogan <peter.co...@gmail.com>
Subject Re: installing pipes on os x
Date Mon, 21 May 2012 17:59:44 GMT

Hi Keith, 

thanks sincerely for the advice and relating your experiences.

Peter Cogan

On 21 May 2012, at 18:57, Keith Wiley wrote:

> Hubris aside (mine I mean), I'm probably not qualified to be the last word on this with
regard to advising you on your project's overall direction, so hopefully others will chime
in on this discussion...but that said, when I started with hadoop a few years ago and our
project exceed the capabilities of direct-Java hadoop interfacing, we briefly entertained
the notion of pipes and learned (to the best of my recollection) that pipes was officially
losing its support within the hadoop community (it may have even been depricated although
I don't remember for sure) and that streaming was being officially advocated as the replacement
(unless I am grossly exaggerating the advice I received at the time).  In fact, the primary
reason I abandoned pipes wasn't so much that the technology itself wouldn't work, but that
there was far less community support for it than for streaming, so whenever I got stuck I
couldn't really ask for help (or I could ask but I would be inquiring a much smaller pool
of potential responders, thus decreasing the scope of any response).
> At any rate, I have been quite satisfied with streaming.  Gotchas have included coding
from a scratch an entire partially abstract C++ class family to offer a streaming interface
to communicate between hadoop and underlying native processing, to say nothing of coding up
binary/ascii conversion in a variety of formats to shuffle data across the i/o streams (for
reasons I can't recall now, the binary streaming interface wasn't a good option for us at
the time).
> Still, all's well.
> Best of luck.
> On May 21, 2012, at 10:26 , Peter Cogan wrote:
>> Hi Keith, 
>> Yeah - maybe that was my first mistake. Since I haven't had much luck getting pipes
working I've started looking at streaming. I am still 'early in my hadoop efforts' so now
is the time for big course corrections
>> thanks
>> Peter
>> On 21 May 2012, at 18:11, Keith Wiley wrote:
>>> Mind if I ask why you prefer pipes to streaming?  I abandoned pipes early in
my hadoop efforts and went all engines full on streaming.
>>> On May 21, 2012, at 04:07 , Peter Cogan wrote:
>>>> Hi,
>>>> I am trying to install pipes on os x following the instructions here
> ________________________________________________________________________________
> Keith Wiley     kwiley@keithwiley.com     keithwiley.com    music.keithwiley.com
> "The easy confidence with which I know another man's religion is folly teaches
> me to suspect that my own is also."
>                                           --  Mark Twain
> ________________________________________________________________________________

View raw message