Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=darose.net; h=message-id:date
	:from:mime-version:to:subject:references:in-reply-to
	:content-type:content-transfer-encoding; q=dns; s=darose.net; b=
	eZ3V2sxzc4R5Q/6BTl+lf7ThQdx3LNudfaqx9PzTGYPLfBMG99at2sFEdUNhkO9Z
	t9GERaQaR72EDTmmWeOYQmLopdZlbvCSUIIkMQIrlRLlMRARcePbqqXFEdgqpyHj
	NYRZYoan+TRguLWn2r8WhDoqGwNU5u7lN0nw7gklPc8=
Message-ID: <4D35E242.20503@darose.net>
Date: Tue, 18 Jan 2011 13:56:02 -0500
From: David Rosenstrauch <darose@darose.net>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.13) Gecko/20101230 Lanikai/3.1.7
MIME-Version: 1.0
To: mapreduce-user@hadoop.apache.org
Subject: Re: how to write custom object using M/R
References: <AANLkTim_r__6YLUniHhs8ms2XaC=Y-WC2Tosa=G5oGpn@mail.gmail.com>
 <24012_1295029670_4D3095A6_24012_35350_1_ECB646CBB62FD044927FDEDF0252D3E701DE10AB978A@THSONEA01CMS03P.one.grp>
In-Reply-To: 
 <24012_1295029670_4D3095A6_24012_35350_1_ECB646CBB62FD044927FDEDF0252D3E701DE10AB978A@THSONEA01CMS03P.one.grp>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

I assumed you were already doing this but yes, Alain is correct, you=20
need to set the output format too.

I initialize writing to sequence files like so:

job.setOutputFormatClass(SequenceFileOutputFormat.class);
FileOutputFormat.setOutputName(job, dataSourceName);
FileOutputFormat.setOutputPath(job, hdfsJobOutputPath);
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class);
SequenceFileOutputFormat.setOutputCompressionType(job,=20
SequenceFile.CompressionType.BLOCK);

DR


On 01/14/2011 01:27 PM, MONTMORY Alain wrote:
> Hi,
>
> I think you have to put :
>              job.setOutputFormatClass(SequenceFileOutputFormat.class);
> to make it works..
> hopes this help
>
> Alain
>
> [@@THALES GROUP RESTRICTED@@]
>
> De : Joan [mailto:joan.monplet@gmail.com]
> Envoy=E9 : vendredi 14 janvier 2011 13:58
> =C0 : mapreduce-user
> Objet : how to write custom object using M/R
>
> Hi,
>
> I'm trying to write (K,V) where K is a Text object and V's CustomObject=
. But It doesn't run.
>
> I'm configuring output job like: SequenceFileInputFormat so I have job =
with:
>
>          job.setMapOutputKeyClass(Text.class);
>          job.setMapOutputValueClass(CustomObject.class);
>          job.setOutputKeyClass(Text.class);
>          job.setOutputValueClass(CustomObject.class);
>
>          SequenceFileOutputFormat.setOutputPath(job, new Path("myPath")=
;
>
> And I obtain the next output (this is a file: part-r-00000):
>
> K  CustomObject@2b237512
> K  CustomObject@24db06de
> ...
>
> When this job finished I run other job which input is SequenceFileInput=
Format but It doesn't run:
>
> The configuration's second job is:
>
>          job.setInputFormatClass(SequenceFileInputFormat.class);
>          SequenceFileInputFormat.addInputPath(job, new Path("myPath"));
>
> But I get an error:
>
> java.io.IOException: hdfs://localhost:30000/user/hadoop/out/part-r-0000=
0 not a SequenceFile
>          at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.=
java:1523)
>          at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFil=
e.java:1483)
>          at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFil=
e.java:1451)
>          at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFil=
e.java:1432)
>          at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordRea=
der.initialize(SequenceFileRecordReader.java:60)
>
>
> Can someone help me? Because I don't understand it. I don't know to sav=
e my object in first M/R and how to use it in second M/R
>
> Thanks
>
> Joan
>
>
>