hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mirko Kämpf <mirko.kae...@gmail.com>
Subject Re: Implementing a custom hadoop key and value - need Help
Date Sun, 03 Nov 2013 12:14:03 GMT
public class MyReducer extends


Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {

    public void reduce(*Iterable<MatrixWritable>*  key,
            Iterable<MatrixWritable> values, Context context){
              for(MatrixWritable c : values){

                System.out.println("print value "+c.toString());

            }

}

Usually a key is only one object, not an Iterable.

To make things more clear:

What is the exact k-v-pair you need in the Reducer?

One matrix is the key, and a set of (two matrices together) are used as
value in the Reducer? What I understood from your question is
KVP<Matrix,Matrix[2]>


*But you wrote also:* I need to emit a 2D double array as key and value
from mapper.
Means, you work with a k-v-pair

KVP<Matrix,Matrix>

There Matrix is 2-D matrix of double values.

I suggest:

1.) Define the MR Data Flow.
2.) Build the custom types.
3.) Test the flow (no computation)
4.) Implement logic / computation








2013/11/3 unmesha sreeveni <unmeshabiju@gmail.com>

> I tried with TwoDArrayWritable too.
>
> but i tried it by emitting only one value.
>
> row = E.length;
> col = E[0].length;
>                      TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>                      DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>                      // set values in myInnerArray
>                      for (int k1 = 0; k1 < row; k1++) {
>                         for(int j1=0;j1< col;j1++){
>                             myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>
>                     }
>                  array.set(myInnerArray);
>                  context.write(clusterNumber, array);
>
>
>
>
>
> this is also not working for me
>
> showing NullPointerException
>
> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
> java.lang.Exception: java.lang.NullPointerException
>     at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.NullPointerException
>     at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>     at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>     at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>     at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>     at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>     at edu.Mapper.map(Mapper.java:277)
>
>
> Mapper.java:277 : context.write(clusterNumber, array);
>
>
>
> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:
>
>> @Amr Shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>>  A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>>  * @param args
>>  */
>> private double[][] value;
>>
>> public MatrixWritable() {
>>     // TODO Auto-generated constructor stub
>>       set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>>     // TODO Auto-generated constructor stub
>>       this.value = value;
>> }
>>
>> public void set(double[][] value) {
>>       this.value = value;
>>  }
>>
>> public double[][] getValue() {
>>         return value;
>>  }
>>
>>  @Override
>>   public void write(DataOutput out) throws IOException {
>>      System.out.println("write");
>>      int row=0;
>>       int col=0;
>>         for(int i=0; i<value.length;i++){
>>             row = value.length;
>>             for(int j=0; j<value[i].length; j++){
>>                 col = value[i].length;
>>             }
>>         }
>>         out.writeInt(row);
>>         out.writeInt(col);
>>
>>         System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>>         for(int i=0;i<row ; i++){
>>             for(int j= 0 ; j< col;j++){
>>
>>                  out.writeDouble(value[i][j]);
>>             }
>>         }
>>         //priting array
>>         for(int vali =0;vali< value.length ;vali ++){
>>             for(int valj = 0;valj <value[0].length;valj++){
>>                 System.out.print(value[vali][valj]+ "\t");
>>             }
>>             System.out.println("");
>>         }
>>
>>   }
>>
>>   @Override
>>   public void readFields(DataInput in) throws IOException {
>>       int row = in.readInt();
>>       int col = in.readInt();
>>
>>       double[][] value = new double[row][col];
>>       for(int i=0;i<row ; i++){
>>             for(int j= 0 ; j< col;j++){
>>                 value[i][j] = in.readDouble();
>>
>>             }
>>         }
>>
>>   }
>>
>>   @Override
>>   public int hashCode() {
>>
>>   }
>>
>>   @Override
>>   public boolean equals(Object o) {
>>
>>   }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>>     // TODO Auto-generated method stub
>>     return 0;
>> }
>>  @Override
>>   public String toString() {
>>
>>     return Arrays.toString(value);
>>
>>   }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only
custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>>     public void reduce(Iterable<MatrixWritable>  key,
>>             Iterable<MatrixWritable> values, Context context){
>>               for(MatrixWritable c : values){
>>
>>                 System.out.println("print value "+c.toString());
>>
>>             }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>>     public String toString() {
>>
>>      String separator = ", ";
>>         StringBuffer result = new StringBuffer();
>>
>>         // iterate over the first dimension
>>         for (int i = 0; i < value.length; i++) {
>>             // iterate over the second dimension
>>             for(int j = 0; j < value[i].length; j++){
>>                 result.append(value[i][j]);
>>                 System.out.print(value[i][j]);
>>                 result.append(separator);
>>             }
>>             // remove the last separator
>>             result.setLength(result.length() - separator.length());
>>             // add a line break.
>>             result.append("\n");
>>         }
>>
>>
>>         return result.toString();
>>
>>
>>   }
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:
>>
>>> @Amr shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>>  A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>>  * @param args
>>>  */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>>     // TODO Auto-generated constructor stub
>>>       set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>>     // TODO Auto-generated constructor stub
>>>       this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>>       this.value = value;
>>>  }
>>>
>>> public double[][] getValue() {
>>>         return value;
>>>  }
>>>
>>>  @Override
>>>   public void write(DataOutput out) throws IOException {
>>>      System.out.println("write");
>>>      int row=0;
>>>       int col=0;
>>>         for(int i=0; i<value.length;i++){
>>>             row = value.length;
>>>             for(int j=0; j<value[i].length; j++){
>>>                 col = value[i].length;
>>>             }
>>>         }
>>>         out.writeInt(row);
>>>         out.writeInt(col);
>>>
>>>         System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>>         for(int i=0;i<row ; i++){
>>>             for(int j= 0 ; j< col;j++){
>>>
>>>                  out.writeDouble(value[i][j]);
>>>             }
>>>         }
>>>         //priting array
>>>         for(int vali =0;vali< value.length ;vali ++){
>>>             for(int valj = 0;valj <value[0].length;valj++){
>>>                 System.out.print(value[vali][valj]+ "\t");
>>>             }
>>>             System.out.println("");
>>>         }
>>>
>>>   }
>>>
>>>   @Override
>>>   public void readFields(DataInput in) throws IOException {
>>>       int row = in.readInt();
>>>       int col = in.readInt();
>>>
>>>       double[][] value = new double[row][col];
>>>       for(int i=0;i<row ; i++){
>>>             for(int j= 0 ; j< col;j++){
>>>                 value[i][j] = in.readDouble();
>>>
>>>             }
>>>         }
>>>
>>>   }
>>>
>>>   @Override
>>>   public int hashCode() {
>>>
>>>   }
>>>
>>>   @Override
>>>   public boolean equals(Object o) {
>>>
>>>   }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>>     // TODO Auto-generated method stub
>>>     return 0;
>>> }
>>>  @Override
>>>   public String toString() {
>>>
>>>     return Arrays.toString(value);
>>>
>>>   }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting
only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>>     public void reduce(Iterable<MatrixWritable>  key,
>>>             Iterable<MatrixWritable> values, Context context){
>>>               for(MatrixWritable c : values){
>>>
>>>                 System.out.println("print value "+c.toString());
>>>
>>>             }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>>     public String toString() {
>>>
>>>      String separator = ", ";
>>>         StringBuffer result = new StringBuffer();
>>>
>>>         // iterate over the first dimension
>>>         for (int i = 0; i < value.length; i++) {
>>>             // iterate over the second dimension
>>>             for(int j = 0; j < value[i].length; j++){
>>>                 result.append(value[i][j]);
>>>                 System.out.print(value[i][j]);
>>>                 result.append(separator);
>>>             }
>>>             // remove the last separator
>>>             result.setLength(result.length() - separator.length());
>>>             // add a line break.
>>>             result.append("\n");
>>>         }
>>>
>>>
>>>         return result.toString();
>>>
>>>
>>>   }
>>>
>>>
>>>
>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <amrnablus@gmail.com> wrote:
>>>
>>>> Can you share the code?
>>>>
>>>> sent from mobile
>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <unmeshabiju@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> thanks Steve Loughran and Amr Shahin
>>>>> Amr Shahin , i refered "
>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>> the same thing only. but my toString is not returning anything.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <amrnablus@gmail.com>wrote:
>>>>>
>>>>>> Check this out:
>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>> It shows how to create a customer writable. If  you have "hadoop
the
>>>>>> definitive guide" there is a really good explanation about custom
data
>>>>>> types.
>>>>>>
>>>>>> Happy Halloween
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>
>>>>>> > this is my post from stackoverflow
>>>>>> > but i am not getting any response.
>>>>>> >
>>>>>> >
>>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>>> are
>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>> > we have to create a custom datatype.but how?I am new to these
custom
>>>>>> > datatypes. i dnt have any idea where to start.I am doing some
of
>>>>>> the matrix
>>>>>> > multiplication in a given dataset.and after that i need to emit
the
>>>>>> value
>>>>>> > of
>>>>>> >  A*Atrns which will be a matrix and Atrans*D which will also
be a
>>>>>> matrix.so
>>>>>> > how to emit these matrices from mapper.And the value should
be
>>>>>> > corresponding to the key itself.I think for that we need to
use
>>>>>> > WritableComparable.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > public class MatrixWritable implements
>>>>>> WritableComparable<MatrixWritable>{
>>>>>> >
>>>>>> > /**
>>>>>> >  * @param args
>>>>>> >  */
>>>>>> > private double[][] value;
>>>>>> >
>>>>>> > public MatrixWritable() {
>>>>>> >     // TODO Auto-generated constructor stub
>>>>>> >       set(new double[0][0]);
>>>>>> > }
>>>>>> >
>>>>>> > public MatrixWritable(double[][] value) {
>>>>>> >     // TODO Auto-generated constructor stub
>>>>>> >       this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public void set(double[][] value) {
>>>>>> >       this.value = value;
>>>>>> >  }
>>>>>> >
>>>>>> > public double[][] getValue() {
>>>>>> >         return value;
>>>>>> >  }
>>>>>> >
>>>>>> >  @Override
>>>>>> >   public void write(DataOutput out) throws IOException {
>>>>>> >      System.out.println("write");
>>>>>> >      int row=0;
>>>>>> >       int col=0;
>>>>>> >         for(int i=0; i<value.length;i++){
>>>>>> >             row = value.length;
>>>>>> >             for(int j=0; j<value[i].length; j++){
>>>>>> >                 col = value[i].length;
>>>>>> >             }
>>>>>> >         }
>>>>>> >         out.writeInt(row);
>>>>>> >         out.writeInt(col);
>>>>>> >
>>>>>> >         System.out.println("\nTotal no of observations:
>>>>>> "+row+":"+col);
>>>>>> >
>>>>>> >         for(int i=0;i<row ; i++){
>>>>>> >             for(int j= 0 ; j< col;j++){
>>>>>> >
>>>>>> >                  out.writeDouble(value[i][j]);
>>>>>> >             }
>>>>>> >         }
>>>>>> >         //priting array
>>>>>> >         for(int vali =0;vali< value.length ;vali ++){
>>>>>> >             for(int valj = 0;valj <value[0].length;valj++){
>>>>>> >                 System.out.print(value[vali][valj]+ "\t");
>>>>>> >             }
>>>>>> >             System.out.println("");
>>>>>> >         }
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >   @Override
>>>>>> >   public void readFields(DataInput in) throws IOException {
>>>>>> >       int row = in.readInt();
>>>>>> >       int col = in.readInt();
>>>>>> >
>>>>>> >       double[][] value = new double[row][col];
>>>>>> >       for(int i=0;i<row ; i++){
>>>>>> >             for(int j= 0 ; j< col;j++){
>>>>>> >                 value[i][j] = in.readDouble();
>>>>>> >
>>>>>> >             }
>>>>>> >         }
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >   @Override
>>>>>> >   public int hashCode() {
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >   @Override
>>>>>> >   public boolean equals(Object o) {
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >
>>>>>> > @Override
>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>> >     // TODO Auto-generated method stub
>>>>>> >     return 0;
>>>>>> > }
>>>>>> >  @Override
>>>>>> >   public String toString() {
>>>>>> >
>>>>>> >     return Arrays.toString(value);
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > I wrote matrix write,readfields and toString.
>>>>>> >
>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>> >
>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>> with
>>>>>> > emiting only custom value- for checking)
>>>>>> >
>>>>>> > public class MyReducer extends
>>>>>> >
>>>>>> >
>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text>
{
>>>>>> >
>>>>>> >     public void reduce(Iterable<MatrixWritable>  key,
>>>>>> >             Iterable<MatrixWritable> values, Context context){
>>>>>> >               for(MatrixWritable c : values){
>>>>>> >
>>>>>> >                 System.out.println("print value "+c.toString());
>>>>>> >
>>>>>> >             }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but Nothing is printing.when i tried to print value[0].length
in
>>>>>> toString()
>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>> wrong.and i
>>>>>> > also needed to print my data asmatrix so i tried
>>>>>> >
>>>>>> >     public String toString() {
>>>>>> >
>>>>>> >      String separator = ", ";
>>>>>> >         StringBuffer result = new StringBuffer();
>>>>>> >
>>>>>> >         // iterate over the first dimension
>>>>>> >         for (int i = 0; i < value.length; i++) {
>>>>>> >             // iterate over the second dimension
>>>>>> >             for(int j = 0; j < value[i].length; j++){
>>>>>> >                 result.append(value[i][j]);
>>>>>> >                 System.out.print(value[i][j]);
>>>>>> >                 result.append(separator);
>>>>>> >             }
>>>>>> >             // remove the last separator
>>>>>> >             result.setLength(result.length() - separator.length());
>>>>>> >             // add a line break.
>>>>>> >             result.append("\n");
>>>>>> >         }
>>>>>> >
>>>>>> >
>>>>>> >         return result.toString();
>>>>>> >
>>>>>> >
>>>>>> >   }
>>>>>> >
>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>> datatype
>>>>>> > CompareTo is neccessary right .
>>>>>> >
>>>>>> > 4.so what should i include in that methods
>>>>>> CompareTo,hashcode,equals and
>>>>>> > what are these methods intended for.
>>>>>> > Any Idea.Pls suggest a solution.
>>>>>> >
>>>>>> > --
>>>>>> > *Thanks & Regards*
>>>>>> > *
>>>>>> > *
>>>>>> > Unmesha Sreeveni U.B*
>>>>>> > *
>>>>>> > *Junior Developer
>>>>>> > *
>>>>>> > *Amrita Center For Cyber Security
>>>>>> > *
>>>>>> > *
>>>>>> > Amritapuri.
>>>>>> >
>>>>>> > www.amrita.edu/cyber/
>>>>>> > *
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>

Mime
View raw message