giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Krause ...@ckrause.org>
Subject Re: Counter limit
Date Sat, 14 Sep 2013 15:40:01 GMT
I'm setting this property and the number of compute threads in the
giraph-site.xml. This works very well. Thanks again for your help.

Cheers,
Christian


2013/9/10 Claudio Martella <claudio.martella@gmail.com>

> you can set it in your giraph-site.xml, but it should work on the command
> line.
>
>
> On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause <me@ckrause.org> wrote:
>
>> I still see the number of counters increasing in the job tracker :(. Can
>> I also set it in my giraph-site.xml or directly in my MasterCompute class?
>>
>> Cheers,
>> Christian
>>
>>
>> 2013/9/10 Claudio Martella <claudio.martella@gmail.com>
>>
>>> one the command line, you can use the -D option after the GiraphRunner
>>> class before the GiraphRunner specific parameters, e.g. -D giraph.
>>> useSuperstepCounters=false
>>>
>>>
>>> On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause <me@ckrause.org>wrote:
>>>
>>>> Thanks a lot. One last question: where do I set options like
>>>> USE_SUPERSTEP_COUNTERS?
>>>>
>>>> Christian
>>>>
>>>>
>>>> 2013/9/9 André Kelpe <efeshundertelf@googlemail.com>
>>>>
>>>>> On older versions of hadoop, you cannot set the counters to a higher
>>>>> value. That was only introduced later. I had this issue on CDH3 (~1.5
>>>>> years ago) and my solution was to disable all counters for the giraph
>>>>> job, to make it work. If you use a more modern version of  hadoop, it
>>>>> should be possible to increase the limit though.
>>>>>
>>>>> - André
>>>>>
>>>>> 2013/9/9 Avery Ching <aching@apache.org>:
>>>>> > If you are running out of counters, you can turn off the superstep
>>>>> counters
>>>>> >
>>>>> >   /** Use superstep counters? (boolean) */
>>>>> >   BooleanConfOption USE_SUPERSTEP_COUNTERS =
>>>>> >       new BooleanConfOption("giraph.useSuperstepCounters", true,
>>>>> >           "Use superstep counters? (boolean)");
>>>>> >
>>>>> >
>>>>> > On 9/9/13 6:43 AM, Claudio Martella wrote:
>>>>> >
>>>>> > No, I used a different counters limit on that hadoop version. Setting
>>>>> > mapreduce.job.counters.limit to a higher number and restarting JT
>>>>> and TT
>>>>> > worked for me. Maybe 64000 might be too high? Try setting it to
512.
>>>>> Does
>>>>> > not look like the case, but who knows.
>>>>> >
>>>>> >
>>>>> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org>
>>>>> wrote:
>>>>> >>
>>>>> >> Sorry, it still doesn't work (I ran into a different problem
before
>>>>> I
>>>>> >> reached the limit).
>>>>> >>
>>>>> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe
>>>>> >> hardcoded?
>>>>> >>
>>>>> >> Cheers
>>>>> >> Christian
>>>>> >>
>>>>> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
>>>>> >>
>>>>> >>> I changed the property name to mapred.job.counters.limit
and
>>>>> restarted it
>>>>> >>> again. Now it works.
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> Christian
>>>>> >>>
>>>>> >>>
>>>>> >>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
>>>>> >>>>
>>>>> >>>> did you restart TT and JT?
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi,
>>>>> >>>>> I've increased the counter limit in mapred-site.xml,
but I still
>>>>> get
>>>>> >>>>> the error: Exceeded counter limits - Counters=121
Limit=120.
>>>>> Groups=6
>>>>> >>>>> Limit=50.
>>>>> >>>>>
>>>>> >>>>> This is my config:
>>>>> >>>>>
>>>>> >>>>>  cat conf/mapred-site.xml
>>>>> >>>>> <?xml version="1.0"?>
>>>>> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>> >>>>>
>>>>> >>>>> <!-- Put site-specific property overrides in
this file. -->
>>>>> >>>>>
>>>>> >>>>> <configuration>
>>>>> >>>>> ...
>>>>> >>>>>     <property>
>>>>> >>>>>         <name>mapreduce.job.counters.limit</name>
>>>>> >>>>>         <value>64000</value>
>>>>> >>>>>     </property>
>>>>> >>>>>     <property>
>>>>> >>>>>         <name>mapred.task.timeout</name>
>>>>> >>>>>         <value>2400000</value>
>>>>> >>>>>     </property>
>>>>> >>>>> ...
>>>>> >>>>> </configuration>
>>>>> >>>>>
>>>>> >>>>> Any ideas?
>>>>> >>>>>
>>>>> >>>>> Cheers,
>>>>> >>>>> Christian
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>>    Claudio Martella
>>>>> >>>>    claudio.martella@gmail.com
>>>>> >>>
>>>>> >>>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> >    Claudio Martella
>>>>> >    claudio.martella@gmail.com
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>    Claudio Martella
>>>    claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>

Mime
View raw message