Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2B2A010B85 for ; Wed, 24 Apr 2013 17:19:26 +0000 (UTC) Received: (qmail 69972 invoked by uid 500); 24 Apr 2013 17:19:20 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 69870 invoked by uid 500); 24 Apr 2013 17:19:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 69862 invoked by uid 99); 24 Apr 2013 17:19:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Apr 2013 17:19:20 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tangxun.alice@gmail.com designates 209.85.215.50 as permitted sender) Received: from [209.85.215.50] (HELO mail-la0-f50.google.com) (209.85.215.50) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Apr 2013 17:19:16 +0000 Received: by mail-la0-f50.google.com with SMTP id fs13so1603515lab.37 for ; Wed, 24 Apr 2013 10:18:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=wTxEvUBX2ckJqhDjlXBLf57/+QR/NahJAF/eYm5Nz1I=; b=nxQI8aG976WEnOX7k4K6Vh9PF6pHclGt7BvHmRRLGEPXRsEa7Z4NNH4ituCYTgvFnC O4kDiH55Bd+Q3Pz3LdCP4cWBQKWi65bLsqBopaSCy9yeBa5k4WxE+jpcnSMe+N2CSslR lvX6DhVev8cF83nIm1bygwkmqAZXZHN+Arw624Kkn7uhex8I73ytCxfq0Irjwwt/sWpk ia1/pHII/vxuApJzk2A33QjITr2b2NiBpCRnvnIl1fpkWqSHIi9sPw2mDt1x/WVLwHD9 YR2BJrALnWUI4skhjxS3QAHftxnahPK48NiWpBjw0G+xcbvO7M+7Mf83ZZc/FnZDmH03 G/Ww== X-Received: by 10.112.20.133 with SMTP id n5mr17680196lbe.113.1366823934554; Wed, 24 Apr 2013 10:18:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.114.161.133 with HTTP; Wed, 24 Apr 2013 10:18:34 -0700 (PDT) In-Reply-To: References: From: Xun TANG Date: Wed, 24 Apr 2013 10:18:34 -0700 Message-ID: Subject: Re: Pipes Property Not Passed In To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae94732d316facb04db1e7fdc X-Virus-Checked: Checked by ClamAV on apache.org --14dae94732d316facb04db1e7fdc Content-Type: text/plain; charset=ISO-8859-1 I think I pinned down the problem, but still can't find a solution. It seems like 'hadoop.pipes.java.recordreader' is not the correct property name to use? Because I tried setting other properties (e.g. hadoop.pipes.executable) in conf.xml and they were effectively passed in to pipes when the program runs. Where could I find the source code of pipes reading conf.xml file? Where could I find a list of all properties (for pipes) and their default values? Best I could find is this http://hadoop.apache.org/docs/r1.1.2/commands_manual.html but it does not list all properties. Any suggestion? Thanks, Alice On Tue, Apr 23, 2013 at 10:14 PM, Xun TANG wrote: > I implemented my own InputFormat/RecordReader, and I try to run it with > Hadoop Pipes. I understand I could pass in properties to Pipes program by > either: > > > hadoop.pipes.java.recordreader > false > > > or alterntively "-D hadoop.pipes.java.recordreader=false". > > However, when I ran with above configuration and with my record reader, I > got error > Hadoop Pipes Exception: RecordReader defined when not needed. at > impl/HadoopPipes.cc:798 in virtual void > HadoopPipes::TaskContextImpl::runMap(std::string, int, bool) > > It pipes did not seem to pick up my setting of > hadoop.pipes.java.recordreader as false. > > I've tried using conf.xml or putting -D or the combine of both. None > worked. I've googled the whole day but could not find the reason. Did I > miss something here? > > I am using hadoop-1.0.4. > > Here is my conf.xml > > > > > hadoop.pipes.executable > bin/cpc > > > hadoop.pipes.java.recordreader > false > > > hadoop.pipes.java.recordwriter > true > > > > Here is the command > > $HADOOP pipes \ > -conf $CONF \ > -files 0 \ > -libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \ > -input $IN \ > -output $OUT \ > -program bin/$NAME \ > -reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \ > -inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat > > where $CONF is full path to conf.xml > > I could provide more info if that hellps to determine the reason. > > --14dae94732d316facb04db1e7fdc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I think I pinned down the problem, but still ca= n't find a solution.=A0

It seems l= ike=A0'hadoop.pipes.java.recordreader' is not the correct = property name to use?
Because I tried setting other p= roperties (e.g.=A0hadoop.pipes.executable) in conf.xml and they were effectively passed in to pipe= s when the program runs.=A0

Where could I find the source code of pipes r= eading conf.xml file? Where could I find a list of all properties (for pipe= s) and their default values? Best I could find is this
but it does not list a= ll properties.

Any suggestion?=A0

Thanks,=A0
Alice
<= /div>


On Tue, = Apr 23, 2013 at 10:14 PM, Xun TANG <tangxun.alice@gmail.com><= /span> wrote:
I implemented my own I= nputFormat/RecordReader, and I try to run it with Hadoop Pipes. I understan= d I could pass in properties to Pipes program by either:

<property>
=A0 =A0 =A0 =A0 <name>hadoop.pipes.java.recordreader</name>
=A0 =A0 =A0 =A0 <value>false</value>
</prop= erty>

or alterntively "-D hadoop.pipes.jav= a.recordreader=3Dfalse".=A0

However, when I ran with above configuration and with m= y record reader, I got error=A0
Hadoop Pipes Exception: RecordRea= der defined when not needed. at impl/HadoopPipes.cc:798 in virtual void Had= oopPipes::TaskContextImpl::runMap(std::string, int, bool)

It pipes did not seem to pick up my setting of hadoop.p= ipes.java.recordreader as false.

I've tried us= ing conf.xml or putting -D or the combine of both. None worked. I've go= ogled the whole day but could not find the reason. Did I miss something her= e?=A0

I am using hadoop-1.0.4.=A0

He= re is my conf.xml

<?xml version=3D"1.0&quo= t;?>
<configuration>
=A0 <property>
=A0 =A0 <name>hadoop.pipes.executable</name>
=A0 =A0 = <value>bin/cpc</value>
=A0 </property>
=A0 <property>
=A0 =A0 <name>hadoop.pipes.java.reco= rdreader</name>
=A0 =A0 <value>false</value>
=A0 </property&g= t;
=A0 <property>
=A0 =A0 <name>hadoop.pipe= s.java.recordwriter</name>
=A0 =A0 <value>true</va= lue>
=A0 </property>
</configuration>

<= /div>
Here is the command

$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/bu= ild/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \<= /div>
-inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat<= /div>

where $CONF is full path to conf.xml

I could provide more info if that hellps to determine the reason.=A0=


--14dae94732d316facb04db1e7fdc--