hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joan <joan.monp...@gmail.com>
Subject Re: how to write custom object using M/R
Date Wed, 19 Jan 2011 14:36:15 GMT
Hi Lance,

My custom object has Writable implement but I don't overrride toString
method?

*public class MyWritable implements DBWritable, Writable, Cloneable  {

    int id;
    String str;

    @Override
    public void readFields(ResultSet rs) throws SQLException {

        id = rs.getInt(1);
        str = rs.getString(2);
    }

    @Override
    public void write(PreparedStatement pstmt) throws SQLException {
        // do nothing
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        id = in.readInt();
        str = Text.readString(in);
    }

    @Override
    public void write(DataOutput out) throws IOException {

        out.writeInt(id);
        Text.writeString(out, str);
    }
}*

But I don't understand why not serialize object,

Thanks

Joan



2011/1/17 Lance Norskog <goksron@gmail.com>

> Does you custom object have Writable implemented? Also, does it have
> toString() implemented? I think this means the Writable code does not
> work:
>
> K  CustomObject@2b237512
> K  CustomObject@24db06de
>
> This is Java's default toString() method.
>
> On Mon, Jan 17, 2011 at 12:19 AM, Joan <joan.monplet@gmail.com> wrote:
> > Hi Alain,
> >
> > I put it, but It didn't work.
> >
> > Joan
> >
> > 2011/1/14 MONTMORY Alain <alain.montmory@thalesgroup.com>
> >>
> >> Hi,
> >>
> >>
> >>
> >> I think you have to put :
> >>
> >>             job.setOutputFormatClass(SequenceFileOutputFormat.class);
> >>
> >> to make it works..
> >>
> >> hopes this help
> >>
> >>
> >>
> >> Alain
> >>
> >>
> >>
> >> [@@THALES GROUP RESTRICTED@@]
> >>
> >>
> >>
> >> De : Joan [mailto:joan.monplet@gmail.com]
> >> Envoyé : vendredi 14 janvier 2011 13:58
> >> À : mapreduce-user
> >> Objet : how to write custom object using M/R
> >>
> >>
> >>
> >> Hi,
> >>
> >> I'm trying to write (K,V) where K is a Text object and V's CustomObject.
> >> But It doesn't run.
> >>
> >> I'm configuring output job like: SequenceFileInputFormat so I have job
> >> with:
> >>
> >>         job.setMapOutputKeyClass(Text.class);
> >>         job.setMapOutputValueClass(CustomObject.class);
> >>         job.setOutputKeyClass(Text.class);
> >>         job.setOutputValueClass(CustomObject.class);
> >>
> >>         SequenceFileOutputFormat.setOutputPath(job, new Path("myPath");
> >>
> >> And I obtain the next output (this is a file: part-r-00000):
> >>
> >> K  CustomObject@2b237512
> >> K  CustomObject@24db06de
> >> ...
> >>
> >> When this job finished I run other job which input is
> >> SequenceFileInputFormat but It doesn't run:
> >>
> >> The configuration's second job is:
> >>
> >>         job.setInputFormatClass(SequenceFileInputFormat.class);
> >>         SequenceFileInputFormat.addInputPath(job, new Path("myPath"));
> >>
> >> But I get an error:
> >>
> >> java.io.IOException: hdfs://localhost:30000/user/hadoop/out/part-r-00000
> >> not a SequenceFile
> >>         at
> >> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1523)
> >>         at
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1483)
> >>         at
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1451)
> >>         at
> >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1432)
> >>         at
> >>
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:60)
> >>
> >>
> >> Can someone help me? Because I don't understand it. I don't know to save
> >> my object in first M/R and how to use it in second M/R
> >>
> >> Thanks
> >>
> >> Joan
> >>
> >>
> >
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Mime
View raw message