giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Young Han <young....@uwaterloo.ca>
Subject Re: SccCompitationTestInMemory - LimitExceededException
Date Wed, 11 Mar 2015 17:50:48 GMT
This seems like the known problem with MapReduce counters. Try adding the
following to your hadoop-*/conf/mapred-site.xml:

  <property>
    <name>mapreduce.job.counters.max</name>
    <value>1000000</value>
  </property>
  <property>
    <name>mapreduce.job.counters.limit</name>
    <value>1000000</value>
  </property>

This does the trick for me on Hadoop 1.0.4, and should work for 0.20 as
well. Not sure about YARN.

Young

On Wed, Mar 11, 2015 at 6:28 AM, Michał Szynkiewicz <szynkiewicz@gmail.com>
wrote:

> I was able to increase the counters limit with: Counters.MAX_COUNTER_LIMIT
> = 2024 (works for hadoop_1 and hadoop 1.2.1).
>
> Then it turned out that whatever limit I set, it is always exceeded.
>
> It turned out that for some reason IntOverwriteAggregator that
> SccPhaseMasterCompute uses to propagate algorithm phase didn't work as
> expected. When read from "copmutations" it had correct value, while read
> from master computation it returned the old value.
>
> I am writing a similar test, where the value to be passed only increases
> and I was able to work around this issue by using Max instead of Overwrite
> aggregator.
>
> Note that I didn't tried to run it yet, these are just results from unit
> tests.
>
> btw, I'm using release-1.1.0
>
>
>
>
> 2015-03-01 23:42 GMT+01:00 Michał Szynkiewicz <szynkiewicz@gmail.com>:
>
>> Hi,
>>
>> I'm trying to run SccComputationTestInMemory and I'm
>> hitting org.apache.hadoop.mapreduce.counters.LimitExceededException: Too
>> many counters: 121 max=120
>>
>> I tried adding both
>> conf.set("mapreduce.job.counters.max", Integer.toString(1024));
>> and
>> conf.set("mapreduce.job.counters.limit", Integer.toString(1024));
>> at the begging of the test, but none of these changed the limit of
>> counters.
>>
>> I tried -Phadoop_2 with hadoop.version=2.6.0 and 2.5.1, -Phadoop_1 with
>> 1.2.1, -Phadoop_0.20.203.
>>
>> How can I run this test successfully?
>>
>> Thanks
>>
>> Michał
>>
>
>

Mime
View raw message