hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das" <d...@yahoo-inc.com>
Subject RE: Counters giving double values
Date Wed, 16 Apr 2008 11:17:37 GMT
Pls file a jira for the counter updates part. It will be excellent if you
can also attach a testcase that can reproduce the problem (maybe a stripped
down version of your app or something). 

> -----Original Message-----
> From: rude@rosa.com [mailto:rude@rosa.com] 
> Sent: Wednesday, April 16, 2008 4:35 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Counters giving double values
> 
> hadoop 0.16.2
> (and as i remember, i had the same issue with 0.16.0)
> 
> Yes, the final data output at the end IS CORRECT. 
> only the counter values are wrong.
> 
> i didnt try to run it in a distributed environment yet. only local.
> 
> 
> 
> 
> 
> 
> "Devaraj Das" <ddas@yahoo-inc.com>
> 04/16/2008 12:56 PM
> Please respond to
> core-user@hadoop.apache.org
> 
> 
> To
> <core-user@hadoop.apache.org>
> cc
> 
> Subject
> RE: Counters giving double values
> 
> 
> 
> 
> 
> 
> Also, in those cases where you see wrong counter values, did 
> you validate
> the final (reduce) output for correctness (I am just trying 
> to see whether
> the problem is with the Counter updates). 
> 
> > -----Original Message-----
> > From: Devaraj Das [mailto:ddas@yahoo-inc.com] 
> > Sent: Wednesday, April 16, 2008 4:23 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > Thanks for the detailed answer. Which hadoop version are you 
> > on? If you are confident that it is not a problem with your 
> > app, pls raise a jira.
> > 
> > 
> >   _____ 
> > 
> > From: rude@rosa.com [mailto:rude@rosa.com]
> > Sent: Wednesday, April 16, 2008 3:25 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > 
> > 
> > Thanks so far. 
> > 
> > key and values are custom implementations. 
> > 
> > key implements WritableComparable 
> > value extends VersionedWritable 
> > 
> > btw. The only problem i encounter is that the Counter values 
> > are wrong. If i
> > check the records in the MapFile (re-read it) which is 
> > written as the output
> > of the mapred job, the amount of records is correct and 
> > represents the halve
> > of the reported counter value. 
> > the same applies for the results of the operations which are 
> > carried out in
> > the reduce(). everything is correct, except the counter values. 
> > 
> > the whole thing happens only sometimes. 
> > 
> > 
> > Key serializing / Deserializing, i guess you want to see this 
> > part of the
> > code: 
> > 
> >         public int language; 
> >         public String term; 
> > 
> >         public void readFields(DataInput in) throws IOException { 
> >                 language = in.readInt(); 
> >                 term =         Text.readString(in).toString(); 
> >         } 
> > 
> >         public void write(DataOutput out) throws IOException { 
> >                 out.writeInt(language); 
> >                 Text.writeString(out, term); 
> >         } 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 


Mime
View raw message