hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xun TANG <tangxun.al...@gmail.com>
Subject Pipes Property Not Passed In
Date Wed, 24 Apr 2013 05:14:28 GMT
I implemented my own InputFormat/RecordReader, and I try to run it with
Hadoop Pipes. I understand I could pass in properties to Pipes program by
either:

<property>
        <name>hadoop.pipes.java.recordreader</name>
        <value>false</value>
</property>

or alterntively "-D hadoop.pipes.java.recordreader=false".

However, when I ran with above configuration and with my record reader, I
got error
Hadoop Pipes Exception: RecordReader defined when not needed. at
impl/HadoopPipes.cc:798 in virtual void
HadoopPipes::TaskContextImpl::runMap(std::string, int, bool)

It pipes did not seem to pick up my setting of
hadoop.pipes.java.recordreader as false.

I've tried using conf.xml or putting -D or the combine of both. None
worked. I've googled the whole day but could not find the reason. Did I
miss something here?

I am using hadoop-1.0.4.

Here is my conf.xml

<?xml version="1.0"?>
<configuration>
  <property>
    <name>hadoop.pipes.executable</name>
    <value>bin/cpc</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordreader</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.pipes.java.recordwriter</name>
    <value>true</value>
  </property>
</configuration>

Here is the command

$HADOOP pipes \
-conf $CONF \
-files 0 \
-libjars $HADOOP_HOME/build/hadoop-test-1.0.4-SNAPSHOT.jar \
-input $IN \
-output $OUT \
-program bin/$NAME \
-reduces 0 -reduce org.apache.hadoop.mapred.lib.IdentityReducer \
-inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat

where $CONF is full path to conf.xml

I could provide more info if that hellps to determine the reason.

Mime
View raw message