giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Krause ...@ckrause.org>
Subject Re: Counter limit
Date Tue, 10 Sep 2013 11:44:14 GMT
I still see the number of counters increasing in the job tracker :(. Can I
also set it in my giraph-site.xml or directly in my MasterCompute class?

Cheers,
Christian


2013/9/10 Claudio Martella <claudio.martella@gmail.com>

> one the command line, you can use the -D option after the GiraphRunner
> class before the GiraphRunner specific parameters, e.g. -D giraph.
> useSuperstepCounters=false
>
>
> On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause <me@ckrause.org> wrote:
>
>> Thanks a lot. One last question: where do I set options like
>> USE_SUPERSTEP_COUNTERS?
>>
>> Christian
>>
>>
>> 2013/9/9 André Kelpe <efeshundertelf@googlemail.com>
>>
>>> On older versions of hadoop, you cannot set the counters to a higher
>>> value. That was only introduced later. I had this issue on CDH3 (~1.5
>>> years ago) and my solution was to disable all counters for the giraph
>>> job, to make it work. If you use a more modern version of  hadoop, it
>>> should be possible to increase the limit though.
>>>
>>> - André
>>>
>>> 2013/9/9 Avery Ching <aching@apache.org>:
>>> > If you are running out of counters, you can turn off the superstep
>>> counters
>>> >
>>> >   /** Use superstep counters? (boolean) */
>>> >   BooleanConfOption USE_SUPERSTEP_COUNTERS =
>>> >       new BooleanConfOption("giraph.useSuperstepCounters", true,
>>> >           "Use superstep counters? (boolean)");
>>> >
>>> >
>>> > On 9/9/13 6:43 AM, Claudio Martella wrote:
>>> >
>>> > No, I used a different counters limit on that hadoop version. Setting
>>> > mapreduce.job.counters.limit to a higher number and restarting JT and
>>> TT
>>> > worked for me. Maybe 64000 might be too high? Try setting it to 512.
>>> Does
>>> > not look like the case, but who knows.
>>> >
>>> >
>>> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org>
>>> wrote:
>>> >>
>>> >> Sorry, it still doesn't work (I ran into a different problem before
I
>>> >> reached the limit).
>>> >>
>>> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe
>>> >> hardcoded?
>>> >>
>>> >> Cheers
>>> >> Christian
>>> >>
>>> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
>>> >>
>>> >>> I changed the property name to mapred.job.counters.limit and
>>> restarted it
>>> >>> again. Now it works.
>>> >>>
>>> >>> Thanks,
>>> >>> Christian
>>> >>>
>>> >>>
>>> >>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
>>> >>>>
>>> >>>> did you restart TT and JT?
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org>
>>> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>> I've increased the counter limit in mapred-site.xml, but
I still
>>> get
>>> >>>>> the error: Exceeded counter limits - Counters=121 Limit=120.
>>> Groups=6
>>> >>>>> Limit=50.
>>> >>>>>
>>> >>>>> This is my config:
>>> >>>>>
>>> >>>>>  cat conf/mapred-site.xml
>>> >>>>> <?xml version="1.0"?>
>>> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>> >>>>>
>>> >>>>> <!-- Put site-specific property overrides in this file.
-->
>>> >>>>>
>>> >>>>> <configuration>
>>> >>>>> ...
>>> >>>>>     <property>
>>> >>>>>         <name>mapreduce.job.counters.limit</name>
>>> >>>>>         <value>64000</value>
>>> >>>>>     </property>
>>> >>>>>     <property>
>>> >>>>>         <name>mapred.task.timeout</name>
>>> >>>>>         <value>2400000</value>
>>> >>>>>     </property>
>>> >>>>> ...
>>> >>>>> </configuration>
>>> >>>>>
>>> >>>>> Any ideas?
>>> >>>>>
>>> >>>>> Cheers,
>>> >>>>> Christian
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>>    Claudio Martella
>>> >>>>    claudio.martella@gmail.com
>>> >>>
>>> >>>
>>> >
>>> >
>>> >
>>> > --
>>> >    Claudio Martella
>>> >    claudio.martella@gmail.com
>>> >
>>> >
>>>
>>
>>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>

Mime
View raw message