hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amr Shahin <amrnab...@gmail.com>
Subject Re: Implementing a custom hadoop key and value - need Help
Date Thu, 31 Oct 2013 12:27:48 GMT
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If  you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.

Happy Halloween


On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:

> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
>  A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
>  * @param args
>  */
> private double[][] value;
>
> public MatrixWritable() {
>     // TODO Auto-generated constructor stub
>       set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
>     // TODO Auto-generated constructor stub
>       this.value = value;
> }
>
> public void set(double[][] value) {
>       this.value = value;
>  }
>
> public double[][] getValue() {
>         return value;
>  }
>
>  @Override
>   public void write(DataOutput out) throws IOException {
>      System.out.println("write");
>      int row=0;
>       int col=0;
>         for(int i=0; i<value.length;i++){
>             row = value.length;
>             for(int j=0; j<value[i].length; j++){
>                 col = value[i].length;
>             }
>         }
>         out.writeInt(row);
>         out.writeInt(col);
>
>         System.out.println("\nTotal no of observations: "+row+":"+col);
>
>         for(int i=0;i<row ; i++){
>             for(int j= 0 ; j< col;j++){
>
>                  out.writeDouble(value[i][j]);
>             }
>         }
>         //priting array
>         for(int vali =0;vali< value.length ;vali ++){
>             for(int valj = 0;valj <value[0].length;valj++){
>                 System.out.print(value[vali][valj]+ "\t");
>             }
>             System.out.println("");
>         }
>
>   }
>
>   @Override
>   public void readFields(DataInput in) throws IOException {
>       int row = in.readInt();
>       int col = in.readInt();
>
>       double[][] value = new double[row][col];
>       for(int i=0;i<row ; i++){
>             for(int j= 0 ; j< col;j++){
>                 value[i][j] = in.readDouble();
>
>             }
>         }
>
>   }
>
>   @Override
>   public int hashCode() {
>
>   }
>
>   @Override
>   public boolean equals(Object o) {
>
>   }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
>     // TODO Auto-generated method stub
>     return 0;
> }
>  @Override
>   public String toString() {
>
>     return Arrays.toString(value);
>
>   }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
>     public void reduce(Iterable<MatrixWritable>  key,
>             Iterable<MatrixWritable> values, Context context){
>               for(MatrixWritable c : values){
>
>                 System.out.println("print value "+c.toString());
>
>             }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
>     public String toString() {
>
>      String separator = ", ";
>         StringBuffer result = new StringBuffer();
>
>         // iterate over the first dimension
>         for (int i = 0; i < value.length; i++) {
>             // iterate over the second dimension
>             for(int j = 0; j < value[i].length; j++){
>                 result.append(value[i][j]);
>                 System.out.print(value[i][j]);
>                 result.append(separator);
>             }
>             // remove the last separator
>             result.setLength(result.length() - separator.length());
>             // add a line break.
>             result.append("\n");
>         }
>
>
>         return result.toString();
>
>
>   }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>

Mime
View raw message