hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <>
Subject Re: Job counters limit exceeded exception
Date Wed, 02 Jan 2013 11:20:05 GMT

These happens when operators are used in queries (Hive Operators). Hive creates 4 counters
per operator, max upto 1000, plus a few additional counters like file read/write, partitions
and tables. Hence the number of counter required is going to be dependent upon the query.

Using "EXPLAIN EXTENDED" and "grep -ri operators | wc -l" print out the used numbers of operators.
Use this value to tweak the MR settings carefully. 

Praveen has a good explanation 'bout counters online:

Rule of thumb for Hive:
count of operators * 4 + n (n for file ops and other stuff).


On Jan 2, 2013, at 10:35 AM, Krishna Rao <> wrote:

> A particular query that I run fails with the following error:
> ***
> Job 18: Map: 2  Reduce: 1   Cumulative CPU: 3.67 sec   HDFS Read: 0 HDFS
> Write: 0 SUCCESS
> Exception in thread "main"
> org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 121 max=120
> ...
> ***
> Googling suggests that I should increase "mapreduce.job.counters.limit".
> And that the number of counters a job uses
> has an effect on the memory used by the JobTracker, so I shouldn't increase
> this number too high.
> Is there a rule of thumb for what this number should be as a function of
> JobTracker memory? That is should I be cautious and
> increase by 5 at a time, or could I just double it?
> Cheers,
> Krishna

Alexander Alten-Lorenz
German Hadoop LinkedIn Group:

View raw message