hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amr Shahin <amrnab...@gmail.com>
Subject Re: Implementing a custom hadoop key and value - need Help
Date Sat, 02 Nov 2013 14:26:20 GMT
Can you share the code?

sent from mobile
On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <unmeshabiju@gmail.com> wrote:

>
> thanks Steve Loughran and Amr Shahin
> Amr Shahin , i refered "
> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
> the same thing only. but my toString is not returning anything.
>
>
>
> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <amrnablus@gmail.com> wrote:
>
>> Check this out:
>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>> It shows how to create a customer writable. If  you have "hadoop the
>> definitive guide" there is a really good explanation about custom data
>> types.
>>
>> Happy Halloween
>>
>>
>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> >wrote:
>>
>> > this is my post from stackoverflow
>> > but i am not getting any response.
>> >
>> >
>> > I need to emit a 2D double array as key and value from mapper.There are
>> > questions posted in stackoverflow. But they are not answered.
>> > we have to create a custom datatype.but how?I am new to these custom
>> > datatypes. i dnt have any idea where to start.I am doing some of the
>> matrix
>> > multiplication in a given dataset.and after that i need to emit the
>> value
>> > of
>> >  A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so
>> > how to emit these matrices from mapper.And the value should be
>> > corresponding to the key itself.I think for that we need to use
>> > WritableComparable.
>> >
>> >
>> >
>> > public class MatrixWritable implements
>> WritableComparable<MatrixWritable>{
>> >
>> > /**
>> >  * @param args
>> >  */
>> > private double[][] value;
>> >
>> > public MatrixWritable() {
>> >     // TODO Auto-generated constructor stub
>> >       set(new double[0][0]);
>> > }
>> >
>> > public MatrixWritable(double[][] value) {
>> >     // TODO Auto-generated constructor stub
>> >       this.value = value;
>> > }
>> >
>> > public void set(double[][] value) {
>> >       this.value = value;
>> >  }
>> >
>> > public double[][] getValue() {
>> >         return value;
>> >  }
>> >
>> >  @Override
>> >   public void write(DataOutput out) throws IOException {
>> >      System.out.println("write");
>> >      int row=0;
>> >       int col=0;
>> >         for(int i=0; i<value.length;i++){
>> >             row = value.length;
>> >             for(int j=0; j<value[i].length; j++){
>> >                 col = value[i].length;
>> >             }
>> >         }
>> >         out.writeInt(row);
>> >         out.writeInt(col);
>> >
>> >         System.out.println("\nTotal no of observations: "+row+":"+col);
>> >
>> >         for(int i=0;i<row ; i++){
>> >             for(int j= 0 ; j< col;j++){
>> >
>> >                  out.writeDouble(value[i][j]);
>> >             }
>> >         }
>> >         //priting array
>> >         for(int vali =0;vali< value.length ;vali ++){
>> >             for(int valj = 0;valj <value[0].length;valj++){
>> >                 System.out.print(value[vali][valj]+ "\t");
>> >             }
>> >             System.out.println("");
>> >         }
>> >
>> >   }
>> >
>> >   @Override
>> >   public void readFields(DataInput in) throws IOException {
>> >       int row = in.readInt();
>> >       int col = in.readInt();
>> >
>> >       double[][] value = new double[row][col];
>> >       for(int i=0;i<row ; i++){
>> >             for(int j= 0 ; j< col;j++){
>> >                 value[i][j] = in.readDouble();
>> >
>> >             }
>> >         }
>> >
>> >   }
>> >
>> >   @Override
>> >   public int hashCode() {
>> >
>> >   }
>> >
>> >   @Override
>> >   public boolean equals(Object o) {
>> >
>> >   }
>> >
>> >
>> > @Override
>> > public int compareTo(MatrixWritable o) {
>> >     // TODO Auto-generated method stub
>> >     return 0;
>> > }
>> >  @Override
>> >   public String toString() {
>> >
>> >     return Arrays.toString(value);
>> >
>> >   }
>> >
>> >
>> >
>> > }
>> >
>> > I wrote matrix write,readfields and toString.
>> >
>> > 1.But my toString is not returning anything. why is it so?
>> >
>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>> > emiting only custom value- for checking)
>> >
>> > public class MyReducer extends
>> >
>> >
>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>> >
>> >     public void reduce(Iterable<MatrixWritable>  key,
>> >             Iterable<MatrixWritable> values, Context context){
>> >               for(MatrixWritable c : values){
>> >
>> >                 System.out.println("print value "+c.toString());
>> >
>> >             }
>> >
>> > }
>> >
>> > but Nothing is printing.when i tried to print value[0].length in
>> toString()
>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i
>> > also needed to print my data asmatrix so i tried
>> >
>> >     public String toString() {
>> >
>> >      String separator = ", ";
>> >         StringBuffer result = new StringBuffer();
>> >
>> >         // iterate over the first dimension
>> >         for (int i = 0; i < value.length; i++) {
>> >             // iterate over the second dimension
>> >             for(int j = 0; j < value[i].length; j++){
>> >                 result.append(value[i][j]);
>> >                 System.out.print(value[i][j]);
>> >                 result.append(separator);
>> >             }
>> >             // remove the last separator
>> >             result.setLength(result.length() - separator.length());
>> >             // add a line break.
>> >             result.append("\n");
>> >         }
>> >
>> >
>> >         return result.toString();
>> >
>> >
>> >   }
>> >
>> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
>> > CompareTo is neccessary right .
>> >
>> > 4.so what should i include in that methods CompareTo,hashcode,equals and
>> > what are these methods intended for.
>> > Any Idea.Pls suggest a solution.
>> >
>> > --
>> > *Thanks & Regards*
>> > *
>> > *
>> > Unmesha Sreeveni U.B*
>> > *
>> > *Junior Developer
>> > *
>> > *Amrita Center For Cyber Security
>> > *
>> > *
>> > Amritapuri.
>> >
>> > www.amrita.edu/cyber/
>> > *
>> >
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>

Mime
View raw message