giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Krause ...@ckrause.org>
Subject Re: Counter limit
Date Tue, 10 Sep 2013 11:15:41 GMT
Thanks a lot. One last question: where do I set options like
USE_SUPERSTEP_COUNTERS?

Christian


2013/9/9 André Kelpe <efeshundertelf@googlemail.com>

> On older versions of hadoop, you cannot set the counters to a higher
> value. That was only introduced later. I had this issue on CDH3 (~1.5
> years ago) and my solution was to disable all counters for the giraph
> job, to make it work. If you use a more modern version of  hadoop, it
> should be possible to increase the limit though.
>
> - André
>
> 2013/9/9 Avery Ching <aching@apache.org>:
> > If you are running out of counters, you can turn off the superstep
> counters
> >
> >   /** Use superstep counters? (boolean) */
> >   BooleanConfOption USE_SUPERSTEP_COUNTERS =
> >       new BooleanConfOption("giraph.useSuperstepCounters", true,
> >           "Use superstep counters? (boolean)");
> >
> >
> > On 9/9/13 6:43 AM, Claudio Martella wrote:
> >
> > No, I used a different counters limit on that hadoop version. Setting
> > mapreduce.job.counters.limit to a higher number and restarting JT and TT
> > worked for me. Maybe 64000 might be too high? Try setting it to 512. Does
> > not look like the case, but who knows.
> >
> >
> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org> wrote:
> >>
> >> Sorry, it still doesn't work (I ran into a different problem before I
> >> reached the limit).
> >>
> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe
> >> hardcoded?
> >>
> >> Cheers
> >> Christian
> >>
> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
> >>
> >>> I changed the property name to mapred.job.counters.limit and restarted
> it
> >>> again. Now it works.
> >>>
> >>> Thanks,
> >>> Christian
> >>>
> >>>
> >>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
> >>>>
> >>>> did you restart TT and JT?
> >>>>
> >>>>
> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org>
> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> I've increased the counter limit in mapred-site.xml, but I still
get
> >>>>> the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6
> >>>>> Limit=50.
> >>>>>
> >>>>> This is my config:
> >>>>>
> >>>>>  cat conf/mapred-site.xml
> >>>>> <?xml version="1.0"?>
> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>
> >>>>> <!-- Put site-specific property overrides in this file. -->
> >>>>>
> >>>>> <configuration>
> >>>>> ...
> >>>>>     <property>
> >>>>>         <name>mapreduce.job.counters.limit</name>
> >>>>>         <value>64000</value>
> >>>>>     </property>
> >>>>>     <property>
> >>>>>         <name>mapred.task.timeout</name>
> >>>>>         <value>2400000</value>
> >>>>>     </property>
> >>>>> ...
> >>>>> </configuration>
> >>>>>
> >>>>> Any ideas?
> >>>>>
> >>>>> Cheers,
> >>>>> Christian
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>    Claudio Martella
> >>>>    claudio.martella@gmail.com
> >>>
> >>>
> >
> >
> >
> > --
> >    Claudio Martella
> >    claudio.martella@gmail.com
> >
> >
>

Mime
View raw message