Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6C02C10882 for ; Sat, 2 Nov 2013 14:27:07 +0000 (UTC) Received: (qmail 50859 invoked by uid 500); 2 Nov 2013 14:26:54 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 50459 invoked by uid 500); 2 Nov 2013 14:26:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50442 invoked by uid 99); 2 Nov 2013 14:26:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Nov 2013 14:26:47 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of amrnablus@gmail.com designates 209.85.215.50 as permitted sender) Received: from [209.85.215.50] (HELO mail-la0-f50.google.com) (209.85.215.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Nov 2013 14:26:41 +0000 Received: by mail-la0-f50.google.com with SMTP id eo20so805592lab.9 for ; Sat, 02 Nov 2013 07:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=YhSomd4Hjf8XAYpe1K8E8Yd6KsYBWdMwmOXAqxqK0lg=; b=dtjwY1Z9nwcoOrwjpEFufzlU3LmptcVg0qr2iSPiz69ULlGZLK8y6NHdb7P8mw6hot cDNTGoS0LS0fOj+wcPhzT4MbDbj1nR26GATaHhgqK2A5oc/6MhmKXjB0Joz0v3jQ0bPL whEAo/2jAqypRfPIVv0xLwBwWrCKcjWBm//cOWRCEPSGY0Hf5atPLas7o2FWckw6PnRm GRl51OxPMiNCICQ2aoNlWrmD3DEg4oVMqqF+JFiUTJC7kSxCh3DeRVXB3IfEFiWWEzwe 9KH3IrUNh1aJjHKg7uGHOWgFvurpCCTa/pphhiDPMtWon9KFv9zMjpxpS/I57SfEnOPV arPQ== MIME-Version: 1.0 X-Received: by 10.112.205.164 with SMTP id lh4mr4305329lbc.15.1383402380764; Sat, 02 Nov 2013 07:26:20 -0700 (PDT) Received: by 10.114.65.105 with HTTP; Sat, 2 Nov 2013 07:26:20 -0700 (PDT) Received: by 10.114.65.105 with HTTP; Sat, 2 Nov 2013 07:26:20 -0700 (PDT) In-Reply-To: References: Date: Sat, 2 Nov 2013 18:26:20 +0400 Message-ID: Subject: Re: Implementing a custom hadoop key and value - need Help From: Amr Shahin To: user Content-Type: multipart/alternative; boundary=001a11c3c87e7cc73604ea3277b0 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3c87e7cc73604ea3277b0 Content-Type: text/plain; charset=ISO-8859-1 Can you share the code? sent from mobile On Nov 1, 2013 7:06 AM, "unmesha sreeveni" wrote: > > thanks Steve Loughran and Amr Shahin > Amr Shahin , i refered " > http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156" > the same thing only. but my toString is not returning anything. > > > > On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin wrote: > >> Check this out: >> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes >> It shows how to create a customer writable. If you have "hadoop the >> definitive guide" there is a really good explanation about custom data >> types. >> >> Happy Halloween >> >> >> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni > >wrote: >> >> > this is my post from stackoverflow >> > but i am not getting any response. >> > >> > >> > I need to emit a 2D double array as key and value from mapper.There are >> > questions posted in stackoverflow. But they are not answered. >> > we have to create a custom datatype.but how?I am new to these custom >> > datatypes. i dnt have any idea where to start.I am doing some of the >> matrix >> > multiplication in a given dataset.and after that i need to emit the >> value >> > of >> > A*Atrns which will be a matrix and Atrans*D which will also be a >> matrix.so >> > how to emit these matrices from mapper.And the value should be >> > corresponding to the key itself.I think for that we need to use >> > WritableComparable. >> > >> > >> > >> > public class MatrixWritable implements >> WritableComparable{ >> > >> > /** >> > * @param args >> > */ >> > private double[][] value; >> > >> > public MatrixWritable() { >> > // TODO Auto-generated constructor stub >> > set(new double[0][0]); >> > } >> > >> > public MatrixWritable(double[][] value) { >> > // TODO Auto-generated constructor stub >> > this.value = value; >> > } >> > >> > public void set(double[][] value) { >> > this.value = value; >> > } >> > >> > public double[][] getValue() { >> > return value; >> > } >> > >> > @Override >> > public void write(DataOutput out) throws IOException { >> > System.out.println("write"); >> > int row=0; >> > int col=0; >> > for(int i=0; i> > row = value.length; >> > for(int j=0; j> > col = value[i].length; >> > } >> > } >> > out.writeInt(row); >> > out.writeInt(col); >> > >> > System.out.println("\nTotal no of observations: "+row+":"+col); >> > >> > for(int i=0;i> > for(int j= 0 ; j< col;j++){ >> > >> > out.writeDouble(value[i][j]); >> > } >> > } >> > //priting array >> > for(int vali =0;vali< value.length ;vali ++){ >> > for(int valj = 0;valj > > System.out.print(value[vali][valj]+ "\t"); >> > } >> > System.out.println(""); >> > } >> > >> > } >> > >> > @Override >> > public void readFields(DataInput in) throws IOException { >> > int row = in.readInt(); >> > int col = in.readInt(); >> > >> > double[][] value = new double[row][col]; >> > for(int i=0;i> > for(int j= 0 ; j< col;j++){ >> > value[i][j] = in.readDouble(); >> > >> > } >> > } >> > >> > } >> > >> > @Override >> > public int hashCode() { >> > >> > } >> > >> > @Override >> > public boolean equals(Object o) { >> > >> > } >> > >> > >> > @Override >> > public int compareTo(MatrixWritable o) { >> > // TODO Auto-generated method stub >> > return 0; >> > } >> > @Override >> > public String toString() { >> > >> > return Arrays.toString(value); >> > >> > } >> > >> > >> > >> > } >> > >> > I wrote matrix write,readfields and toString. >> > >> > 1.But my toString is not returning anything. why is it so? >> > >> > 2.How to print these values with in Reducer ?I tried doing(tried with >> > emiting only custom value- for checking) >> > >> > public class MyReducer extends >> > >> > >> > Reducer { >> > >> > public void reduce(Iterable key, >> > Iterable values, Context context){ >> > for(MatrixWritable c : values){ >> > >> > System.out.println("print value "+c.toString()); >> > >> > } >> > >> > } >> > >> > but Nothing is printing.when i tried to print value[0].length in >> toString() >> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and >> i >> > also needed to print my data asmatrix so i tried >> > >> > public String toString() { >> > >> > String separator = ", "; >> > StringBuffer result = new StringBuffer(); >> > >> > // iterate over the first dimension >> > for (int i = 0; i < value.length; i++) { >> > // iterate over the second dimension >> > for(int j = 0; j < value[i].length; j++){ >> > result.append(value[i][j]); >> > System.out.print(value[i][j]); >> > result.append(separator); >> > } >> > // remove the last separator >> > result.setLength(result.length() - separator.length()); >> > // add a line break. >> > result.append("\n"); >> > } >> > >> > >> > return result.toString(); >> > >> > >> > } >> > >> > Again my output is empty. 3.Inorder to emit a key too as custom datatype >> > CompareTo is neccessary right . >> > >> > 4.so what should i include in that methods CompareTo,hashcode,equals and >> > what are these methods intended for. >> > Any Idea.Pls suggest a solution. >> > >> > -- >> > *Thanks & Regards* >> > * >> > * >> > Unmesha Sreeveni U.B* >> > * >> > *Junior Developer >> > * >> > *Amrita Center For Cyber Security >> > * >> > * >> > Amritapuri. >> > >> > www.amrita.edu/cyber/ >> > * >> > >> > > > > -- > *Thanks & Regards* > > Unmesha Sreeveni U.B > > *Junior Developer* > > *Amrita Center For Cyber Security * > > > * Amritapuri.www.amrita.edu/cyber/ * > --001a11c3c87e7cc73604ea3277b0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Can you share the code?

sent from mobile

On Nov 1, 2013 7:06 AM, "unmesha sreeveni&q= uot; <unmeshabiju@gmail.com= > wrote:

thanks=A0Steve Loughran and=A0Amr Shahin
Amr Shahin , i refered "http://my.safaribookson= line.com/book/databases/hadoop/9780596521974/serialization/id3548156" the same thing only. but my toString is not return= ing anything.



On Thu,= Oct 31, 2013 at 5:57 PM, Amr Shahin <amrnablus@gmail.com>= wrote:
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module= 5.html#writable-notes
It shows how to create a customer writable. If =A0you have "hadoop the=
definitive guide" there is a really good explanation about custom data=
types.

Happy Halloween


On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:

> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There ar= e
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom > datatypes. i dnt have any idea where to start.I am doing some of the m= atrix
> multiplication in a given dataset.and after that i need to emit the va= lue
> of
> =A0A*Atrns which will be a matrix and Atrans*D which will also be a ma= trix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWri= table>{
>
> /**
> =A0* @param args
> =A0*/
> private double[][] value;
>
> public MatrixWritable() {
> =A0 =A0 // TODO Auto-generated constructor stub
> =A0 =A0 =A0 set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> =A0 =A0 // TODO Auto-generated constructor stub
> =A0 =A0 =A0 this.value =3D value;
> }
>
> public void set(double[][] value) {
> =A0 =A0 =A0 this.value =3D value;
> =A0}
>
> public double[][] getValue() {
> =A0 =A0 =A0 =A0 return value;
> =A0}
>
> =A0@Override
> =A0 public void write(DataOutput out) throws IOException {
> =A0 =A0 =A0System.out.println("write");
> =A0 =A0 =A0int row=3D0;
> =A0 =A0 =A0 int col=3D0;
> =A0 =A0 =A0 =A0 for(int i=3D0; i<value.length;i++){
> =A0 =A0 =A0 =A0 =A0 =A0 row =3D value.length;
> =A0 =A0 =A0 =A0 =A0 =A0 for(int j=3D0; j<value[i].length; j++){
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 col =3D value[i].length;
> =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 out.writeInt(row);
> =A0 =A0 =A0 =A0 out.writeInt(col);
>
> =A0 =A0 =A0 =A0 System.out.println("\nTotal no of observations: &= quot;+row+":"+col);
>
> =A0 =A0 =A0 =A0 for(int i=3D0;i<row ; i++){
> =A0 =A0 =A0 =A0 =A0 =A0 for(int j=3D 0 ; j< col;j++){
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0out.writeDouble(value[i][j]);
> =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 //priting array
> =A0 =A0 =A0 =A0 for(int vali =3D0;vali< value.length ;vali ++){
> =A0 =A0 =A0 =A0 =A0 =A0 for(int valj =3D 0;valj <value[0].length;va= lj++){
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 System.out.print(value[vali][valj]+ &q= uot;\t");
> =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 =A0 =A0 System.out.println("");
> =A0 =A0 =A0 =A0 }
>
> =A0 }
>
> =A0 @Override
> =A0 public void readFields(DataInput in) throws IOException {
> =A0 =A0 =A0 int row =3D in.readInt();
> =A0 =A0 =A0 int col =3D in.readInt();
>
> =A0 =A0 =A0 double[][] value =3D new double[row][col];
> =A0 =A0 =A0 for(int i=3D0;i<row ; i++){
> =A0 =A0 =A0 =A0 =A0 =A0 for(int j=3D 0 ; j< col;j++){
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 value[i][j] =3D in.readDouble();
>
> =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 }
>
> =A0 }
>
> =A0 @Override
> =A0 public int hashCode() {
>
> =A0 }
>
> =A0 @Override
> =A0 public boolean equals(Object o) {
>
> =A0 }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> =A0 =A0 // TODO Auto-generated method stub
> =A0 =A0 return 0;
> }
> =A0@Override
> =A0 public String toString() {
>
> =A0 =A0 return Arrays.toString(value);
>
> =A0 }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with<= br> > emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> =A0 =A0 public void reduce(Iterable<MatrixWritable> =A0key,
> =A0 =A0 =A0 =A0 =A0 =A0 Iterable<MatrixWritable> values, Context= context){
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 for(MatrixWritable c : values){
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 System.out.println("print value &= quot;+c.toString());
>
> =A0 =A0 =A0 =A0 =A0 =A0 }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toStr= ing()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.an= d i
> also needed to print my data asmatrix so i tried
>
> =A0 =A0 public String toString() {
>
> =A0 =A0 =A0String separator =3D ", ";
> =A0 =A0 =A0 =A0 StringBuffer result =3D new StringBuffer();
>
> =A0 =A0 =A0 =A0 // iterate over the first dimension
> =A0 =A0 =A0 =A0 for (int i =3D 0; i < value.length; i++) {
> =A0 =A0 =A0 =A0 =A0 =A0 // iterate over the second dimension
> =A0 =A0 =A0 =A0 =A0 =A0 for(int j =3D 0; j < value[i].length; j++){=
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 result.append(value[i][j]);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 System.out.print(value[i][j]);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 result.append(separator);
> =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 =A0 =A0 // remove the last separator
> =A0 =A0 =A0 =A0 =A0 =A0 result.setLength(result.length() - separator.l= ength());
> =A0 =A0 =A0 =A0 =A0 =A0 // add a line break.
> =A0 =A0 =A0 =A0 =A0 =A0 result.append("\n");
> =A0 =A0 =A0 =A0 }
>
>
> =A0 =A0 =A0 =A0 return result.toString();
>
>
> =A0 }
>
> Again my output is empty. 3.Inorder to emit a key too as custom dataty= pe
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals a= nd
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.= edu/cyber/
> *
>



--
Thanks & Regards

Unmesha Sreeveni U.B
Junior Developer
Amrita Center For Cyber Security
--001a11c3c87e7cc73604ea3277b0--