hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@rosa.com
Subject RE: Counters giving double values
Date Thu, 17 Apr 2008 13:39:28 GMT
hi devraj,

so, i researched the topic with the counters further with some success.
for one i can reproduce it now with a Test.

i am waiting for the password for my JIRA account to get started there - 
somehow i didnt get the password after registration, i sent a mail to 
owen.
i am not familiar with the proceedings on JIRA / ASF... so if you see 
mistakes please guide me. thanks. 

if you are interested, try out the attached testcase. i figured it is some 
timing issue within the localjobrunner (in my case, dont know about 
distributed running yet)  and i will try to provide a patch.
can i now submit this testcase trough JIRA? do i need to change something 
on it?

kind regards,

ud







"Devaraj Das" <ddas@yahoo-inc.com> 
04/16/2008 01:18 PM
Please respond to
core-user@hadoop.apache.org


To
<core-user@hadoop.apache.org>
cc

Subject
RE: Counters giving double values






Pls file a jira for the counter updates part. It will be excellent if you
can also attach a testcase that can reproduce the problem (maybe a 
stripped
down version of your app or something). 

> -----Original Message-----
> From: rude@rosa.com [mailto:rude@rosa.com] 
> Sent: Wednesday, April 16, 2008 4:35 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Counters giving double values
> 
> hadoop 0.16.2
> (and as i remember, i had the same issue with 0.16.0)
> 
> Yes, the final data output at the end IS CORRECT. 
> only the counter values are wrong.
> 
> i didnt try to run it in a distributed environment yet. only local.
> 
> 
> 
> 
> 
> 
> "Devaraj Das" <ddas@yahoo-inc.com>
> 04/16/2008 12:56 PM
> Please respond to
> core-user@hadoop.apache.org
> 
> 
> To
> <core-user@hadoop.apache.org>
> cc
> 
> Subject
> RE: Counters giving double values
> 
> 
> 
> 
> 
> 
> Also, in those cases where you see wrong counter values, did 
> you validate
> the final (reduce) output for correctness (I am just trying 
> to see whether
> the problem is with the Counter updates). 
> 
> > -----Original Message-----
> > From: Devaraj Das [mailto:ddas@yahoo-inc.com] 
> > Sent: Wednesday, April 16, 2008 4:23 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > Thanks for the detailed answer. Which hadoop version are you 
> > on? If you are confident that it is not a problem with your 
> > app, pls raise a jira.
> > 
> > 
> >   _____ 
> > 
> > From: rude@rosa.com [mailto:rude@rosa.com]
> > Sent: Wednesday, April 16, 2008 3:25 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > 
> > 
> > Thanks so far. 
> > 
> > key and values are custom implementations. 
> > 
> > key implements WritableComparable 
> > value extends VersionedWritable 
> > 
> > btw. The only problem i encounter is that the Counter values 
> > are wrong. If i
> > check the records in the MapFile (re-read it) which is 
> > written as the output
> > of the mapred job, the amount of records is correct and 
> > represents the halve
> > of the reported counter value. 
> > the same applies for the results of the operations which are 
> > carried out in
> > the reduce(). everything is correct, except the counter values. 
> > 
> > the whole thing happens only sometimes. 
> > 
> > 
> > Key serializing / Deserializing, i guess you want to see this 
> > part of the
> > code: 
> > 
> >         public int language; 
> >         public String term; 
> > 
> >         public void readFields(DataInput in) throws IOException { 
> >                 language = in.readInt(); 
> >                 term =         Text.readString(in).toString(); 
> >         } 
> > 
> >         public void write(DataOutput out) throws IOException { 
> >                 out.writeInt(language); 
> >                 Text.writeString(out, term); 
> >         } 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 



Mime
View raw message