giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Kelpe <efeshundert...@googlemail.com>
Subject Re: Counter limit
Date Mon, 09 Sep 2013 18:32:27 GMT
On older versions of hadoop, you cannot set the counters to a higher
value. That was only introduced later. I had this issue on CDH3 (~1.5
years ago) and my solution was to disable all counters for the giraph
job, to make it work. If you use a more modern version of  hadoop, it
should be possible to increase the limit though.

- André

2013/9/9 Avery Ching <aching@apache.org>:
> If you are running out of counters, you can turn off the superstep counters
>
>   /** Use superstep counters? (boolean) */
>   BooleanConfOption USE_SUPERSTEP_COUNTERS =
>       new BooleanConfOption("giraph.useSuperstepCounters", true,
>           "Use superstep counters? (boolean)");
>
>
> On 9/9/13 6:43 AM, Claudio Martella wrote:
>
> No, I used a different counters limit on that hadoop version. Setting
> mapreduce.job.counters.limit to a higher number and restarting JT and TT
> worked for me. Maybe 64000 might be too high? Try setting it to 512. Does
> not look like the case, but who knows.
>
>
> On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org> wrote:
>>
>> Sorry, it still doesn't work (I ran into a different problem before I
>> reached the limit).
>>
>> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe
>> hardcoded?
>>
>> Cheers
>> Christian
>>
>> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
>>
>>> I changed the property name to mapred.job.counters.limit and restarted it
>>> again. Now it works.
>>>
>>> Thanks,
>>> Christian
>>>
>>>
>>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
>>>>
>>>> did you restart TT and JT?
>>>>
>>>>
>>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org> wrote:
>>>>>
>>>>> Hi,
>>>>> I've increased the counter limit in mapred-site.xml, but I still get
>>>>> the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6
>>>>> Limit=50.
>>>>>
>>>>> This is my config:
>>>>>
>>>>>  cat conf/mapred-site.xml
>>>>> <?xml version="1.0"?>
>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>
>>>>> <!-- Put site-specific property overrides in this file. -->
>>>>>
>>>>> <configuration>
>>>>> ...
>>>>>     <property>
>>>>>         <name>mapreduce.job.counters.limit</name>
>>>>>         <value>64000</value>
>>>>>     </property>
>>>>>     <property>
>>>>>         <name>mapred.task.timeout</name>
>>>>>         <value>2400000</value>
>>>>>     </property>
>>>>> ...
>>>>> </configuration>
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Cheers,
>>>>> Christian
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>    Claudio Martella
>>>>    claudio.martella@gmail.com
>>>
>>>
>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>
>

Mime
View raw message