hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Implementing VectorWritable
Date Tue, 29 Dec 2009 05:55:01 GMT
The readFields and write method is empty ?

When data is transfered from map phase to reduce phase, data is serialized
and deserialized , so the write and readFields will be called. You should
not leave them empty.


Jeff Zhang


On Tue, Dec 29, 2009 at 1:29 PM, bharath v <
bharathvissapragada1990@gmail.com> wrote:

> Hi ,
>
> I've implemented a simple VectorWritable class as follows
>
>
> package com;
>
> import org.apache.hadoop.*;
> import org.apache.hadoop.io.*;
> import java.io.*;
> import java.util.Vector;
>
>
> public class VectorWritable implements WritableComparable {
>  private Vector<String> value = new Vector();
>
>  public VectorWritable() {}
>
>  public VectorWritable(Vector<String> value) { set(value); }
>
>  public void set(Vector<String> val) { this.value = val;
>  }
>
>  public Vector<String> get() { return this.value; }
>
>  public void readFields(DataInput in) throws IOException {
>    //value = in.readInt();
>  }
>
>  public void write(DataOutput out) throws IOException {
>  //  out.writeInt(value);
>  }
>
>  public boolean equals(Object o) {
>    if (!(o instanceof VectorWritable))
>      return false;
>    VectorWritable other = (VectorWritable)o;
>    return this.value.equals(other.value);
>  }
>
>  public int hashCode() {
>    return value.hashCode();
>  }
>
>  public int compareTo(Object o) {
>    Vector thisValue = this.value;
>    Vector thatValue = ((VectorWritable)o).value;
>    return (thisValue.size()<thatValue.size() ? -1 :
> (thisValue.size()==thatValue.size() ? 0 : 1));
>  }
>
>  public String toString() {
>    return value.toString();
>  }
>
>  public static class Comparator extends WritableComparator {
>    public Comparator() {
>      super(VectorWritable.class);
>    }
>
>    public int compare(byte[] b1, int s1, int l1,
>                       byte[] b2, int s2, int l2) {
>
>      int thisValue = readInt(b1, s1);
>      int thatValue = readInt(b2, s2);
>      return (thisValue<thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
>    }
>  }
>
>  static {                                        // register this
> comparator
>    WritableComparator.define(VectorWritable.class, new Comparator());
>  }
> }
>
> The map phase is outputting correct <Text,VectorWritable> pairs .. but in
> reduce phase
> when I iterate over the values Iterable.. Iam getting the size of the
> vector
> to be 0; I think there is a minor
> mistake in my VectorWritable Implementation .. Can anyone point it..
>
> Thanks
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message