hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@rosa.com
Subject RE: Counters giving double values
Date Wed, 16 Apr 2008 11:04:32 GMT
hadoop 0.16.2
(and as i remember, i had the same issue with 0.16.0)

Yes, the final data output at the end IS CORRECT. 
only the counter values are wrong.

i didnt try to run it in a distributed environment yet. only local.






"Devaraj Das" <ddas@yahoo-inc.com> 
04/16/2008 12:56 PM
Please respond to
core-user@hadoop.apache.org


To
<core-user@hadoop.apache.org>
cc

Subject
RE: Counters giving double values






Also, in those cases where you see wrong counter values, did you validate
the final (reduce) output for correctness (I am just trying to see whether
the problem is with the Counter updates). 

> -----Original Message-----
> From: Devaraj Das [mailto:ddas@yahoo-inc.com] 
> Sent: Wednesday, April 16, 2008 4:23 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Counters giving double values
> 
> Thanks for the detailed answer. Which hadoop version are you 
> on? If you are confident that it is not a problem with your 
> app, pls raise a jira.
> 
> 
>   _____ 
> 
> From: rude@rosa.com [mailto:rude@rosa.com]
> Sent: Wednesday, April 16, 2008 3:25 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Counters giving double values
> 
> 
> 
> Thanks so far. 
> 
> key and values are custom implementations. 
> 
> key implements WritableComparable 
> value extends VersionedWritable 
> 
> btw. The only problem i encounter is that the Counter values 
> are wrong. If i
> check the records in the MapFile (re-read it) which is 
> written as the output
> of the mapred job, the amount of records is correct and 
> represents the halve
> of the reported counter value. 
> the same applies for the results of the operations which are 
> carried out in
> the reduce(). everything is correct, except the counter values. 
> 
> the whole thing happens only sometimes. 
> 
> 
> Key serializing / Deserializing, i guess you want to see this 
> part of the
> code: 
> 
>         public int language; 
>         public String term; 
> 
>         public void readFields(DataInput in) throws IOException { 
>                 language = in.readInt(); 
>                 term =         Text.readString(in).toString(); 
>         } 
> 
>         public void write(DataOutput out) throws IOException { 
>                 out.writeInt(language); 
>                 Text.writeString(out, term); 
>         } 
> 
> 
> 
> 
> 



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message