hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gianluigi Zanetti <gianluigi.zane...@crs4.it>
Subject Re: C++ pipes on full (nonpseudo) cluster
Date Tue, 30 Mar 2010 20:43:37 GMT
Hello.
Did you try following the tutorial in 
http://wiki.apache.org/hadoop/C++WordCount ?

We use C++ pipes in production on a large cluster, and it works.

--gianluigi


On Tue, 2010-03-30 at 13:28 -0700, Keith Wiley wrote:
> No responses yet, although I admit it's only been a few hours.
> 
> As a follow-up, permit me to pose the following question:
> 
> Is it, in fact, impossible to run C++ pipes on a fully-distributed system (as opposed
to a pseudo-distributed system)?  I haven't found any definitive clarification on this topic
one way or the other.  The only statement that I found in the least bit illuminating is in
the O'Reilly book (not official Hadoop documentation mind you), p.38, which states:
> 
> "To run a Pipes job, we need to run Hadoop in pseudo-distributed mode...Pipes doesn't
run in standalone (local) mode, since it relies on Hadoop's distributed cache mechanism, which
works only when HDFS is running."
> 
> The phrasing of those statements is a little unclear in that the distinction being made
appears to be between standalone and pseudo-distributed mode, without any specific reference
to fully-distributed mode.  Namely, the section that qualifies the need for pseudo-distributed
mode (the need for HDFS) would obviously also apply to full distributed mode despite the lack
of mention of fully distributed mode in the quoted section.  So can pipes run in fully distributed
mode or not?
> 
> Bottom line, I can't get C++ pipes to work on a fully distributed cluster yet and I don't
know if I am wasting my time, if this is a truly impossible effort or if it can be done and
I simply haven't figured out how to do it yet.
> 
> Thanks for any help.
> 
> ________________________________________________________________________________
> Keith Wiley               kwiley@keithwiley.com               www.keithwiley.com
> 
> "The easy confidence with which I know another man's religion is folly teaches
> me to suspect that my own is also."
>   -- Mark Twain
> ________________________________________________________________________________
> 
> 
> 

Mime
View raw message