hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruv Kumar <dku...@ecs.umass.edu>
Subject Re: ArrayWritable usage
Date Mon, 04 Jul 2011 20:24:27 GMT
Thanks,  your answer and the documentation at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/ArrayWritable.htmlfixed
the issue.

For others benefit, I have reproduced the solution below.

Modified test code:

   SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new
Path("/home/input/in"), LongWritable.class, IntArrayWritable.class);

      IntArrayWritable array = new IntArrayWritable();

      IntWritable[] ints = new IntWritable[4];

      for (int i =0 ; i < 4; i++) {
        ints[i] = new IntWritable(i);

      }

      array.set(ints);

      writer.append(new LongWritable(1), array);

      writer.close();



New IntArrayWritableClass:

public class IntArrayWritable extends ArrayWritable {
  public IntArrayWritable() {
    super(IntWritable.class);
  }
}


On Mon, Jul 4, 2011 at 3:00 PM, Joey Echeverria <joey@cloudera.com> wrote:

> ArrayWritable doesn't serialize type information. You need to subclass it
> (e.g. IntArrayWritable) and create a no arg constructor which calls
> super(IntWritable.class).
>
> Use this instead of ArrayWritable directly. If you want to store more than
> one type, look at the source for MapWritable to see how it generates type
> codes.
>
> -Joey
> On Jul 4, 2011 2:55 PM, "Dhruv Kumar" <dkumar@ecs.umass.edu> wrote:
> > I'm having some difficulty with using ArrayWritable in the following test
> > code:
> >
> > ArrayWritable array = new ArrayWritable(IntWritable.class);
> >
> > IntWritable[] ints = new IntWritable[4];
> >
> > for (int i =0 ; i < 4; i++) {
> > ints[i] = new IntWritable(i);
> >
> > }
> >
> > array.set(ints);
> >
> > writer.append(new LongWritable(1), array);
> >
> > writer.close();
> >
> > After starting the map reduce application's test, I'm getting the
> following
> > errors which I suspect are due to improper use of ArrayWritable. Can
> someone
> > tell me if the code given above is correct?
> >
> > java.lang.RuntimeException: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.<init>()
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> > at
> >
>
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> > at
> >
>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
> > at
> > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> > at org.apache.hadoop.mapred.Child.main(Child.java:253)
> > Caused by: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.<init>()
> > at java.lang.Class.getConstructor0(Class.java:2706)
> > at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> > ... 15 more
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message