hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Implementing a custom hadoop key and value - need Help
Date Thu, 31 Oct 2013 11:03:07 GMT
this is my post from stackoverflow
but i am not getting any response.


I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
 A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.



public class MatrixWritable implements WritableComparable<MatrixWritable>{

/**
 * @param args
 */
private double[][] value;

public MatrixWritable() {
    // TODO Auto-generated constructor stub
      set(new double[0][0]);
}

public MatrixWritable(double[][] value) {
    // TODO Auto-generated constructor stub
      this.value = value;
}

public void set(double[][] value) {
      this.value = value;
 }

public double[][] getValue() {
        return value;
 }

 @Override
  public void write(DataOutput out) throws IOException {
     System.out.println("write");
     int row=0;
      int col=0;
        for(int i=0; i<value.length;i++){
            row = value.length;
            for(int j=0; j<value[i].length; j++){
                col = value[i].length;
            }
        }
        out.writeInt(row);
        out.writeInt(col);

        System.out.println("\nTotal no of observations: "+row+":"+col);

        for(int i=0;i<row ; i++){
            for(int j= 0 ; j< col;j++){

                 out.writeDouble(value[i][j]);
            }
        }
        //priting array
        for(int vali =0;vali< value.length ;vali ++){
            for(int valj = 0;valj <value[0].length;valj++){
                System.out.print(value[vali][valj]+ "\t");
            }
            System.out.println("");
        }

  }

  @Override
  public void readFields(DataInput in) throws IOException {
      int row = in.readInt();
      int col = in.readInt();

      double[][] value = new double[row][col];
      for(int i=0;i<row ; i++){
            for(int j= 0 ; j< col;j++){
                value[i][j] = in.readDouble();

            }
        }

  }

  @Override
  public int hashCode() {

  }

  @Override
  public boolean equals(Object o) {

  }


@Override
public int compareTo(MatrixWritable o) {
    // TODO Auto-generated method stub
    return 0;
}
 @Override
  public String toString() {

    return Arrays.toString(value);

  }



}

I wrote matrix write,readfields and toString.

1.But my toString is not returning anything. why is it so?

2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)

public class MyReducer extends


Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {

    public void reduce(Iterable<MatrixWritable>  key,
            Iterable<MatrixWritable> values, Context context){
              for(MatrixWritable c : values){

                System.out.println("print value "+c.toString());

            }

}

but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried

    public String toString() {

     String separator = ", ";
        StringBuffer result = new StringBuffer();

        // iterate over the first dimension
        for (int i = 0; i < value.length; i++) {
            // iterate over the second dimension
            for(int j = 0; j < value[i].length; j++){
                result.append(value[i][j]);
                System.out.print(value[i][j]);
                result.append(separator);
            }
            // remove the last separator
            result.setLength(result.length() - separator.length());
            // add a line break.
            result.append("\n");
        }


        return result.toString();


  }

Again my output is empty. 3.Inorder to emit a key too as custom datatype
CompareTo is neccessary right .

4.so what should i include in that methods CompareTo,hashcode,equals and
what are these methods intended for.
Any Idea.Pls suggest a solution.

-- 
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.

www.amrita.edu/cyber/
*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message