hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: What is the cost of using a counter
Date Thu, 08 Dec 2011 19:22:25 GMT
Looking at the code all of the instances of Counter for the increment method just to a value
+= incr.  They are not going to be much more expensive then keeping the counter yourself internally.
 What happens in that at periodic intervals the counters are then sent to the JT (pre YARN)
or the ApplicationMaster (in YARN) where it keeps track of them.  It should really have no
impact on how often you increment them.

--Bobby Evans

On 12/8/11 12:48 PM, "Steve Lewis" <lordjoe2000@gmail.com> wrote:

I have several counters that I maintain to allow me to keep statistics on critical operations.
I have my code incrementing the counters in an inner loop
partly to make sure my job is not killed for not making progress. It would be very easy to
keep an internal counter and increment the Hadoop value less frequently.  Assuming
I am currently incrementing a counter several million times in a reduce task- is this costing
me performance and would I be better off incrementing less frequently

Mime
View raw message