giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Counter limit
Date Tue, 10 Sep 2013 14:03:55 GMT
you can set it in your giraph-site.xml, but it should work on the command
line.


On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause <me@ckrause.org> wrote:

> I still see the number of counters increasing in the job tracker :(. Can I
> also set it in my giraph-site.xml or directly in my MasterCompute class?
>
> Cheers,
> Christian
>
>
> 2013/9/10 Claudio Martella <claudio.martella@gmail.com>
>
>> one the command line, you can use the -D option after the GiraphRunner
>> class before the GiraphRunner specific parameters, e.g. -D giraph.
>> useSuperstepCounters=false
>>
>>
>> On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause <me@ckrause.org> wrote:
>>
>>> Thanks a lot. One last question: where do I set options like
>>> USE_SUPERSTEP_COUNTERS?
>>>
>>> Christian
>>>
>>>
>>> 2013/9/9 André Kelpe <efeshundertelf@googlemail.com>
>>>
>>>> On older versions of hadoop, you cannot set the counters to a higher
>>>> value. That was only introduced later. I had this issue on CDH3 (~1.5
>>>> years ago) and my solution was to disable all counters for the giraph
>>>> job, to make it work. If you use a more modern version of  hadoop, it
>>>> should be possible to increase the limit though.
>>>>
>>>> - André
>>>>
>>>> 2013/9/9 Avery Ching <aching@apache.org>:
>>>> > If you are running out of counters, you can turn off the superstep
>>>> counters
>>>> >
>>>> >   /** Use superstep counters? (boolean) */
>>>> >   BooleanConfOption USE_SUPERSTEP_COUNTERS =
>>>> >       new BooleanConfOption("giraph.useSuperstepCounters", true,
>>>> >           "Use superstep counters? (boolean)");
>>>> >
>>>> >
>>>> > On 9/9/13 6:43 AM, Claudio Martella wrote:
>>>> >
>>>> > No, I used a different counters limit on that hadoop version. Setting
>>>> > mapreduce.job.counters.limit to a higher number and restarting JT and
>>>> TT
>>>> > worked for me. Maybe 64000 might be too high? Try setting it to 512.
>>>> Does
>>>> > not look like the case, but who knows.
>>>> >
>>>> >
>>>> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org>
>>>> wrote:
>>>> >>
>>>> >> Sorry, it still doesn't work (I ran into a different problem before
I
>>>> >> reached the limit).
>>>> >>
>>>> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe
>>>> >> hardcoded?
>>>> >>
>>>> >> Cheers
>>>> >> Christian
>>>> >>
>>>> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
>>>> >>
>>>> >>> I changed the property name to mapred.job.counters.limit and
>>>> restarted it
>>>> >>> again. Now it works.
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Christian
>>>> >>>
>>>> >>>
>>>> >>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
>>>> >>>>
>>>> >>>> did you restart TT and JT?
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Hi,
>>>> >>>>> I've increased the counter limit in mapred-site.xml,
but I still
>>>> get
>>>> >>>>> the error: Exceeded counter limits - Counters=121 Limit=120.
>>>> Groups=6
>>>> >>>>> Limit=50.
>>>> >>>>>
>>>> >>>>> This is my config:
>>>> >>>>>
>>>> >>>>>  cat conf/mapred-site.xml
>>>> >>>>> <?xml version="1.0"?>
>>>> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>> >>>>>
>>>> >>>>> <!-- Put site-specific property overrides in this
file. -->
>>>> >>>>>
>>>> >>>>> <configuration>
>>>> >>>>> ...
>>>> >>>>>     <property>
>>>> >>>>>         <name>mapreduce.job.counters.limit</name>
>>>> >>>>>         <value>64000</value>
>>>> >>>>>     </property>
>>>> >>>>>     <property>
>>>> >>>>>         <name>mapred.task.timeout</name>
>>>> >>>>>         <value>2400000</value>
>>>> >>>>>     </property>
>>>> >>>>> ...
>>>> >>>>> </configuration>
>>>> >>>>>
>>>> >>>>> Any ideas?
>>>> >>>>>
>>>> >>>>> Cheers,
>>>> >>>>> Christian
>>>> >>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>>    Claudio Martella
>>>> >>>>    claudio.martella@gmail.com
>>>> >>>
>>>> >>>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >    Claudio Martella
>>>> >    claudio.martella@gmail.com
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>>
>
>


-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message