hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das" <d...@yahoo-inc.com>
Subject RE: Counters giving double values
Date Thu, 17 Apr 2008 16:21:57 GMT
Thanks! Will take a look at the jira issue 3267


  _____  

From: rude@rosa.com [mailto:rude@rosa.com] 
Sent: Thursday, April 17, 2008 7:09 PM
To: core-user@hadoop.apache.org
Subject: RE: Counters giving double values



hi devraj, 

so, i researched the topic with the counters further with some success. 
for one i can reproduce it now with a Test. 

i am waiting for the password for my JIRA account to get started there -
somehow i didnt get the password after registration, i sent a mail to owen. 
i am not familiar with the proceedings on JIRA / ASF... so if you see
mistakes please guide me. thanks. 

if you are interested, try out the attached testcase. i figured it is some
timing issue within the localjobrunner (in my case, dont know about
distributed running yet)  and i will try to provide a patch. 
can i now submit this testcase trough JIRA? do i need to change something on
it? 

kind regards, 

ud 







"Devaraj Das" <ddas@yahoo-inc.com> 


04/16/2008 01:18 PM 


Please respond to
core-user@hadoop.apache.org



To
<core-user@hadoop.apache.org> 

cc

Subject
RE: Counters giving double values

	




Pls file a jira for the counter updates part. It will be excellent if you
can also attach a testcase that can reproduce the problem (maybe a stripped
down version of your app or something). 

> -----Original Message-----
> From: rude@rosa.com [ <mailto:rude@rosa.com> mailto:rude@rosa.com] 
> Sent: Wednesday, April 16, 2008 4:35 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Counters giving double values
> 
> hadoop 0.16.2
> (and as i remember, i had the same issue with 0.16.0)
> 
> Yes, the final data output at the end IS CORRECT. 
> only the counter values are wrong.
> 
> i didnt try to run it in a distributed environment yet. only local.
> 
> 
> 
> 
> 
> 
> "Devaraj Das" <ddas@yahoo-inc.com>
> 04/16/2008 12:56 PM
> Please respond to
> core-user@hadoop.apache.org
> 
> 
> To
> <core-user@hadoop.apache.org>
> cc
> 
> Subject
> RE: Counters giving double values
> 
> 
> 
> 
> 
> 
> Also, in those cases where you see wrong counter values, did 
> you validate
> the final (reduce) output for correctness (I am just trying 
> to see whether
> the problem is with the Counter updates). 
> 
> > -----Original Message-----
> > From: Devaraj Das [ <mailto:ddas@yahoo-inc.com>
mailto:ddas@yahoo-inc.com] 
> > Sent: Wednesday, April 16, 2008 4:23 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > Thanks for the detailed answer. Which hadoop version are you 
> > on? If you are confident that it is not a problem with your 
> > app, pls raise a jira.
> > 
> > 
> >   _____ 
> > 
> > From: rude@rosa.com [ <mailto:rude@rosa.com> mailto:rude@rosa.com]
> > Sent: Wednesday, April 16, 2008 3:25 PM
> > To: core-user@hadoop.apache.org
> > Subject: RE: Counters giving double values
> > 
> > 
> > 
> > Thanks so far. 
> > 
> > key and values are custom implementations. 
> > 
> > key implements WritableComparable 
> > value extends VersionedWritable 
> > 
> > btw. The only problem i encounter is that the Counter values 
> > are wrong. If i
> > check the records in the MapFile (re-read it) which is 
> > written as the output
> > of the mapred job, the amount of records is correct and 
> > represents the halve
> > of the reported counter value. 
> > the same applies for the results of the operations which are 
> > carried out in
> > the reduce(). everything is correct, except the counter values. 
> > 
> > the whole thing happens only sometimes. 
> > 
> > 
> > Key serializing / Deserializing, i guess you want to see this 
> > part of the
> > code: 
> > 
> >         public int language; 
> >         public String term; 
> > 
> >         public void readFields(DataInput in) throws IOException { 
> >                 language = in.readInt(); 
> >                 term =         Text.readString(in).toString(); 
> >         } 
> > 
> >         public void write(DataOutput out) throws IOException { 
> >                 out.writeInt(language); 
> >                 Text.writeString(out, term); 
> >         } 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 





Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message