Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of hadoop.viks@gmail.com
 designates 209.85.215.176 as permitted sender)
MIME-Version: 1.0
Date: Tue, 13 Sep 2011 12:27:54 -0400
Message-ID: 
 <CAHNak6YGy6naQOB5xYwOB+xNs7gHjaY5_BTj_5ZJJZpDD47Myw@mail.gmail.com>
Subject: Outputformat and RecordWriter in Hadoop Pipes
From: Vivek K <hadoop.viks@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307c9bba2416f704acd520dc

--20cf307c9bba2416f704acd520dc
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I am trying to build a Hadoop/MR application in c++ using hadoop-pipes. I
have been able to successfully work with my own mappers and reducers, but
now I need to generate output (from reducer) in a format different from the
default TextOutputFormat. I have a few questions:

(1) Similar to Hadoop streaming, is there an option to set OutputFormat in
HadoopPipes (in order to use say org.apache.hadoop.io.SequenceFile.Writer) ?
I am using Hadoop version 0.20.2.

(2) For a simple test on how to use an in-built non-default writer, I tried
the following:

     hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=false -input input.seq -output output
-inputformat org.apache.hadoop.mapred.SequenceFileInputFormat -writer
org.apache.hadoop.io.SequenceFile.Writer -program my_test_program

     However this fails with a ClassNotFound exception. And if I remove the
-writer flag and use the default writer, it works just fine.

(3) Is there some example or discussion related to how to write your own
RecordWriter and run it with Hadoop-pipes ?

Thanks.

Best,
Vivek
--

--20cf307c9bba2416f704acd520dc--