Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 668A910AB5 for ; Tue, 10 Sep 2013 11:16:32 +0000 (UTC) Received: (qmail 77918 invoked by uid 500); 10 Sep 2013 11:16:32 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 77726 invoked by uid 500); 10 Sep 2013 11:16:29 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 77712 invoked by uid 99); 10 Sep 2013 11:16:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 11:16:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.216.169] (HELO mail-qc0-f169.google.com) (209.85.216.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 11:16:24 +0000 Received: by mail-qc0-f169.google.com with SMTP id c9so1970356qcz.14 for ; Tue, 10 Sep 2013 04:15:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=QQXCCg6WbLZpo7D4rVXHcNK4k8dalPaFzHfA7P7YTuw=; b=OM71qaXU3w0k9aErHsvBEs9ffD1ioVsua/zxcqbiZyb/7dyZH+f+ej0j3OYmMgwlWp nc5EmtOSoZ3B6CtbjlXLdS2e9tsAXR0vOs4zfxilFscQXnmTwRvd5VDI/ag77Qyz0n5v SwZR9wg+5yDRpIWco/9Rt74Y557vLl2cJlHOGjNU+H2XyLe1tpZOSJCMg9JrY8eaK9cB fnRz4CNLknVALHF5Hbc3SXr79xv1TF3C/knIYRROsyYM3kxvTqyCIsks4YS7KYpAPxFr ehlnErFTTB2opXSwgia9EY3Mu/0U9dzJTP/hXJFjMPGS3jzGJEJtWDdV4mGEqPgL9kKu pLrA== X-Gm-Message-State: ALoCoQkkv2PVilsyL6mMvu9GO/6L+rkKt23PqMUzP+Ph28x9ddfqPHjuvf43Si3uqNygMtuJM5ck X-Received: by 10.49.50.7 with SMTP id y7mr30779901qen.45.1378811743064; Tue, 10 Sep 2013 04:15:43 -0700 (PDT) Received: from mail-qc0-x229.google.com (mail-qc0-x229.google.com [2607:f8b0:400d:c01::229]) by mx.google.com with ESMTPSA id a2sm34015680qek.7.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 10 Sep 2013 04:15:42 -0700 (PDT) Received: by mail-qc0-f169.google.com with SMTP id c9so1966308qcz.0 for ; Tue, 10 Sep 2013 04:15:41 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.229.79.70 with SMTP id o6mr19660147qck.21.1378811741510; Tue, 10 Sep 2013 04:15:41 -0700 (PDT) Received: by 10.49.128.40 with HTTP; Tue, 10 Sep 2013 04:15:41 -0700 (PDT) In-Reply-To: References: <522E0B20.4010907@apache.org> Date: Tue, 10 Sep 2013 13:15:41 +0200 Message-ID: Subject: Re: Counter limit From: Christian Krause To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=001a1133beb610ddc804e605a0de X-Virus-Checked: Checked by ClamAV on apache.org --001a1133beb610ddc804e605a0de Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 Andr=E9 Kelpe > On older versions of hadoop, you cannot set the counters to a higher > value. That was only introduced later. I had this issue on CDH3 (~1.5 > years ago) and my solution was to disable all counters for the giraph > job, to make it work. If you use a more modern version of hadoop, it > should be possible to increase the limit though. > > - Andr=E9 > > 2013/9/9 Avery Ching : > > If you are running out of counters, you can turn off the superstep > counters > > > > /** Use superstep counters? (boolean) */ > > BooleanConfOption USE_SUPERSTEP_COUNTERS =3D > > new BooleanConfOption("giraph.useSuperstepCounters", true, > > "Use superstep counters? (boolean)"); > > > > > > On 9/9/13 6:43 AM, Claudio Martella wrote: > > > > No, I used a different counters limit on that hadoop version. Setting > > mapreduce.job.counters.limit to a higher number and restarting JT and T= T > > worked for me. Maybe 64000 might be too high? Try setting it to 512. Do= es > > not look like the case, but who knows. > > > > > > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause wrote= : > >> > >> Sorry, it still doesn't work (I ran into a different problem before I > >> reached the limit). > >> > >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe > >> hardcoded? > >> > >> Cheers > >> Christian > >> > >> Am 09.09.2013 08:29 schrieb "Christian Krause" : > >> > >>> I changed the property name to mapred.job.counters.limit and restarte= d > it > >>> again. Now it works. > >>> > >>> Thanks, > >>> Christian > >>> > >>> > >>> 2013/9/7 Claudio Martella > >>>> > >>>> did you restart TT and JT? > >>>> > >>>> > >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause > wrote: > >>>>> > >>>>> Hi, > >>>>> I've increased the counter limit in mapred-site.xml, but I still ge= t > >>>>> the error: Exceeded counter limits - Counters=3D121 Limit=3D120. Gr= oups=3D6 > >>>>> Limit=3D50. > >>>>> > >>>>> This is my config: > >>>>> > >>>>> cat conf/mapred-site.xml > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ... > >>>>> > >>>>> mapreduce.job.counters.limit > >>>>> 64000 > >>>>> > >>>>> > >>>>> mapred.task.timeout > >>>>> 2400000 > >>>>> > >>>>> ... > >>>>> > >>>>> > >>>>> Any ideas? > >>>>> > >>>>> Cheers, > >>>>> Christian > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Claudio Martella > >>>> claudio.martella@gmail.com > >>> > >>> > > > > > > > > -- > > Claudio Martella > > claudio.martella@gmail.com > > > > > --001a1133beb610ddc804e605a0de Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks a lot. One last question: where do I set options li= ke USE_SUPERSTEP_COUNTERS?

Christian


2013/9/9 Andr=E9 Kelp= e <efeshundertelf@googlemail.com>
On older versions of hadoop, you cannot set = the counters to a higher
value. That was only introduced later. I had this issue on CDH3 (~1.5
years ago) and my solution was to disable all counters for the giraph
job, to make it work. If you use a more modern version of =A0hadoop, it
should be possible to increase the limit though.

- Andr=E9

2013/9/9 Avery Ching <aching@apache= .org>:
> If you are running out of coun= ters, you can turn off the superstep counters
>
> =A0 /** Use superstep counters? (boolean) */
> =A0 BooleanConfOption USE_SUPERSTEP_COUNTERS =3D
> =A0 =A0 =A0 new BooleanConfOption("giraph.useSuperstepCounters&qu= ot;, true,
> =A0 =A0 =A0 =A0 =A0 "Use superstep counters? (boolean)"); >
>
> On 9/9/13 6:43 AM, Claudio Martella wrote:
>
> No, I used a different counters limit on that hadoop version. Setting<= br> > mapreduce.job.counters.limit to a higher number and restarting JT and = TT
> worked for me. Maybe 64000 might be too high? Try setting it to 512. D= oes
> not look like the case, but who knows.
>
>
> On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <me@ckrause.org> wrote:
>>
>> Sorry, it still doesn't work (I ran into a different problem b= efore I
>> reached the limit).
>>
>> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe >> hardcoded?
>>
>> Cheers
>> Christian
>>
>> Am 09.09.2013 08:29 schrieb "Christian Krause" <me@ckrause.org>:
>>
>>> I changed the property name to mapred.job.counters.limit and r= estarted it
>>> again. Now it works.
>>>
>>> Thanks,
>>> Christian
>>>
>>>
>>> 2013/9/7 Claudio Martella <claudio.martella@gmail.com>
>>>>
>>>> did you restart TT and JT?
>>>>
>>>>
>>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <me@ckrause.org> wrote:
>>>>>
>>>>> Hi,
>>>>> I've increased the counter limit in mapred-site.xm= l, but I still get
>>>>> the error: Exceeded counter limits - Counters=3D121 Li= mit=3D120. Groups=3D6
>>>>> Limit=3D50.
>>>>>
>>>>> This is my config:
>>>>>
>>>>> =A0cat conf/mapred-site.xml
>>>>> <?xml version=3D"1.0"?>
>>>>> <?xml-stylesheet type=3D"text/xsl" href= =3D"configuration.xsl"?>
>>>>>
>>>>> <!-- Put site-specific property overrides in this f= ile. -->
>>>>>
>>>>> <configuration>
>>>>> ...
>>>>> =A0 =A0 <property>
>>>>> =A0 =A0 =A0 =A0 <name>mapreduce.job.counters.lim= it</name>
>>>>> =A0 =A0 =A0 =A0 <value>64000</value>
>>>>> =A0 =A0 </property>
>>>>> =A0 =A0 <property>
>>>>> =A0 =A0 =A0 =A0 <name>mapred.task.timeout</na= me>
>>>>> =A0 =A0 =A0 =A0 <value>2400000</value>
>>>>> =A0 =A0 </property>
>>>>> ...
>>>>> </configuration>
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Cheers,
>>>>> Christian
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> =A0 =A0Claudio Martella
>>>> =A0 =A0claud= io.martella@gmail.com
>>>
>>>
>
>
>
> --
> =A0 =A0Claudio Martella
> =A0 =A0claudio.martella@= gmail.com
>
>

--001a1133beb610ddc804e605a0de--