hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Re: Implementing a custom hadoop key and value - need Help
Date Sun, 03 Nov 2013 11:34:54 GMT
@Amr shahin
this is my post from stackoverflow
but i am not getting any response.


I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
 A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.



public class MatrixWritable implements WritableComparable<MatrixWritable>{

/**
 * @param args
 */
private double[][] value;

public MatrixWritable() {
    // TODO Auto-generated constructor stub
      set(new double[0][0]);
}

public MatrixWritable(double[][] value) {
    // TODO Auto-generated constructor stub
      this.value = value;
}

public void set(double[][] value) {
      this.value = value;
 }

public double[][] getValue() {
        return value;
 }

 @Override
  public void write(DataOutput out) throws IOException {
     System.out.println("write");
     int row=0;
      int col=0;
        for(int i=0; i<value.length;i++){
            row = value.length;
            for(int j=0; j<value[i].length; j++){
                col = value[i].length;
            }
        }
        out.writeInt(row);
        out.writeInt(col);

        System.out.println("\nTotal no of observations: "+row+":"+col);

        for(int i=0;i<row ; i++){
            for(int j= 0 ; j< col;j++){

                 out.writeDouble(value[i][j]);
            }
        }
        //priting array
        for(int vali =0;vali< value.length ;vali ++){
            for(int valj = 0;valj <value[0].length;valj++){
                System.out.print(value[vali][valj]+ "\t");
            }
            System.out.println("");
        }

  }

  @Override
  public void readFields(DataInput in) throws IOException {
      int row = in.readInt();
      int col = in.readInt();

      double[][] value = new double[row][col];
      for(int i=0;i<row ; i++){
            for(int j= 0 ; j< col;j++){
                value[i][j] = in.readDouble();

            }
        }

  }

  @Override
  public int hashCode() {

  }

  @Override
  public boolean equals(Object o) {

  }


@Override
public int compareTo(MatrixWritable o) {
    // TODO Auto-generated method stub
    return 0;
}
 @Override
  public String toString() {

    return Arrays.toString(value);

  }



}

I wrote matrix write,readfields and toString.

1.But my toString is not returning anything. why is it so?

2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)

public class MyReducer extends


Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {

    public void reduce(Iterable<MatrixWritable>  key,
            Iterable<MatrixWritable> values, Context context){
              for(MatrixWritable c : values){

                System.out.println("print value "+c.toString());

            }

}

but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried

    public String toString() {

     String separator = ", ";
        StringBuffer result = new StringBuffer();

        // iterate over the first dimension
        for (int i = 0; i < value.length; i++) {
            // iterate over the second dimension
            for(int j = 0; j < value[i].length; j++){
                result.append(value[i][j]);
                System.out.print(value[i][j]);
                result.append(separator);
            }
            // remove the last separator
            result.setLength(result.length() - separator.length());
            // add a line break.
            result.append("\n");
        }


        return result.toString();


  }



On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <amrnablus@gmail.com> wrote:

> Can you share the code?
>
> sent from mobile
> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <unmeshabiju@gmail.com> wrote:
>
>>
>> thanks Steve Loughran and Amr Shahin
>> Amr Shahin , i refered "
>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>> the same thing only. but my toString is not returning anything.
>>
>>
>>
>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <amrnablus@gmail.com> wrote:
>>
>>> Check this out:
>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>> It shows how to create a customer writable. If  you have "hadoop the
>>> definitive guide" there is a really good explanation about custom data
>>> types.
>>>
>>> Happy Halloween
>>>
>>>
>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>> >wrote:
>>>
>>> > this is my post from stackoverflow
>>> > but i am not getting any response.
>>> >
>>> >
>>> > I need to emit a 2D double array as key and value from mapper.There are
>>> > questions posted in stackoverflow. But they are not answered.
>>> > we have to create a custom datatype.but how?I am new to these custom
>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>> matrix
>>> > multiplication in a given dataset.and after that i need to emit the
>>> value
>>> > of
>>> >  A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so
>>> > how to emit these matrices from mapper.And the value should be
>>> > corresponding to the key itself.I think for that we need to use
>>> > WritableComparable.
>>> >
>>> >
>>> >
>>> > public class MatrixWritable implements
>>> WritableComparable<MatrixWritable>{
>>> >
>>> > /**
>>> >  * @param args
>>> >  */
>>> > private double[][] value;
>>> >
>>> > public MatrixWritable() {
>>> >     // TODO Auto-generated constructor stub
>>> >       set(new double[0][0]);
>>> > }
>>> >
>>> > public MatrixWritable(double[][] value) {
>>> >     // TODO Auto-generated constructor stub
>>> >       this.value = value;
>>> > }
>>> >
>>> > public void set(double[][] value) {
>>> >       this.value = value;
>>> >  }
>>> >
>>> > public double[][] getValue() {
>>> >         return value;
>>> >  }
>>> >
>>> >  @Override
>>> >   public void write(DataOutput out) throws IOException {
>>> >      System.out.println("write");
>>> >      int row=0;
>>> >       int col=0;
>>> >         for(int i=0; i<value.length;i++){
>>> >             row = value.length;
>>> >             for(int j=0; j<value[i].length; j++){
>>> >                 col = value[i].length;
>>> >             }
>>> >         }
>>> >         out.writeInt(row);
>>> >         out.writeInt(col);
>>> >
>>> >         System.out.println("\nTotal no of observations: "+row+":"+col);
>>> >
>>> >         for(int i=0;i<row ; i++){
>>> >             for(int j= 0 ; j< col;j++){
>>> >
>>> >                  out.writeDouble(value[i][j]);
>>> >             }
>>> >         }
>>> >         //priting array
>>> >         for(int vali =0;vali< value.length ;vali ++){
>>> >             for(int valj = 0;valj <value[0].length;valj++){
>>> >                 System.out.print(value[vali][valj]+ "\t");
>>> >             }
>>> >             System.out.println("");
>>> >         }
>>> >
>>> >   }
>>> >
>>> >   @Override
>>> >   public void readFields(DataInput in) throws IOException {
>>> >       int row = in.readInt();
>>> >       int col = in.readInt();
>>> >
>>> >       double[][] value = new double[row][col];
>>> >       for(int i=0;i<row ; i++){
>>> >             for(int j= 0 ; j< col;j++){
>>> >                 value[i][j] = in.readDouble();
>>> >
>>> >             }
>>> >         }
>>> >
>>> >   }
>>> >
>>> >   @Override
>>> >   public int hashCode() {
>>> >
>>> >   }
>>> >
>>> >   @Override
>>> >   public boolean equals(Object o) {
>>> >
>>> >   }
>>> >
>>> >
>>> > @Override
>>> > public int compareTo(MatrixWritable o) {
>>> >     // TODO Auto-generated method stub
>>> >     return 0;
>>> > }
>>> >  @Override
>>> >   public String toString() {
>>> >
>>> >     return Arrays.toString(value);
>>> >
>>> >   }
>>> >
>>> >
>>> >
>>> > }
>>> >
>>> > I wrote matrix write,readfields and toString.
>>> >
>>> > 1.But my toString is not returning anything. why is it so?
>>> >
>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>> > emiting only custom value- for checking)
>>> >
>>> > public class MyReducer extends
>>> >
>>> >
>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>> >
>>> >     public void reduce(Iterable<MatrixWritable>  key,
>>> >             Iterable<MatrixWritable> values, Context context){
>>> >               for(MatrixWritable c : values){
>>> >
>>> >                 System.out.println("print value "+c.toString());
>>> >
>>> >             }
>>> >
>>> > }
>>> >
>>> > but Nothing is printing.when i tried to print value[0].length in
>>> toString()
>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>> wrong.and i
>>> > also needed to print my data asmatrix so i tried
>>> >
>>> >     public String toString() {
>>> >
>>> >      String separator = ", ";
>>> >         StringBuffer result = new StringBuffer();
>>> >
>>> >         // iterate over the first dimension
>>> >         for (int i = 0; i < value.length; i++) {
>>> >             // iterate over the second dimension
>>> >             for(int j = 0; j < value[i].length; j++){
>>> >                 result.append(value[i][j]);
>>> >                 System.out.print(value[i][j]);
>>> >                 result.append(separator);
>>> >             }
>>> >             // remove the last separator
>>> >             result.setLength(result.length() - separator.length());
>>> >             // add a line break.
>>> >             result.append("\n");
>>> >         }
>>> >
>>> >
>>> >         return result.toString();
>>> >
>>> >
>>> >   }
>>> >
>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>> datatype
>>> > CompareTo is neccessary right .
>>> >
>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>> and
>>> > what are these methods intended for.
>>> > Any Idea.Pls suggest a solution.
>>> >
>>> > --
>>> > *Thanks & Regards*
>>> > *
>>> > *
>>> > Unmesha Sreeveni U.B*
>>> > *
>>> > *Junior Developer
>>> > *
>>> > *Amrita Center For Cyber Security
>>> > *
>>> > *
>>> > Amritapuri.
>>> >
>>> > www.amrita.edu/cyber/
>>> > *
>>> >
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>


-- 
*Thanks & Regards*

Unmesha Sreeveni U.B

*Junior Developer*

*Amrita Center For Cyber Security*


* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*

Mime
View raw message