Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C387E1058B for ; Mon, 9 Sep 2013 23:22:32 +0000 (UTC) Received: (qmail 74465 invoked by uid 500); 9 Sep 2013 23:22:32 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 74431 invoked by uid 500); 9 Sep 2013 23:22:32 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 74423 invoked by uid 99); 9 Sep 2013 23:22:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Sep 2013 23:22:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of claudio.martella@gmail.com designates 209.85.128.180 as permitted sender) Received: from [209.85.128.180] (HELO mail-ve0-f180.google.com) (209.85.128.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Sep 2013 23:22:25 +0000 Received: by mail-ve0-f180.google.com with SMTP id jz11so4090579veb.11 for ; Mon, 09 Sep 2013 16:22:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=GS3lRjLNx40a6d5ZusRAtzWIwYXtf6uiRQckTIh4ZY0=; b=CH7WKCmtomj46OMFkfnLTuq5yOTsd3qxGmtARs6VFwl+4uZxqpvwXT/CnuXmD+3LFx ujRfbH+VZ5jyrMXbK+NFrUu7Vnrb2LJOzJ0p0oD3V2wZTv0vIbr8dkpRVZDcsj5HTg62 lXWDnRyZWfoeVorHB8qgGn7Ei51dCbCfpIdlv5pfawL6aJWu5m73dpGOW0LaxGte04PU H5zZtY+RVK8wexXefi9gusTaWK02IeMtgHSmF4CJOd+Q0zSBEc1Imz7MSrGqPtFR5auP 4iwImsUNR6Q6fDV1lLry+XoVOQKghTJrsljRQYcwqTx5NpSjVVTeuKHU0MQSzOELR7BF NmtA== X-Received: by 10.52.34.109 with SMTP id y13mr17060134vdi.8.1378768925131; Mon, 09 Sep 2013 16:22:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.141.206 with HTTP; Mon, 9 Sep 2013 16:21:45 -0700 (PDT) In-Reply-To: References: From: Claudio Martella Date: Tue, 10 Sep 2013 01:21:45 +0200 Message-ID: Subject: Re: Out of core execution has no effect on GC crash To: "user@giraph.apache.org" Content-Type: multipart/alternative; boundary=20cf30780c5c028f7404e5fba87d X-Virus-Checked: Checked by ClamAV on apache.org --20cf30780c5c028f7404e5fba87d Content-Type: text/plain; charset=ISO-8859-1 did you extend the heap available to the mapper tasks? e.g. through mapred.child.java.opts. On Tue, Sep 10, 2013 at 12:50 AM, Alexander Asplund wrote: > Thanks for the reply. > > I tried setting giraph.maxPartitionsInMemory to 1, but I'm still > getting OOM: GC limit exceeded. > > Are there any particular cases the OOC will not be able to handle, or > is it supposed to work in all cases? If the latter, it might be that I > have made some configuration error. > > I do have one concern that might indicateI have done something wrong: > to allow OOC to activate without crashing I had to modify the trunk > code. This was because Giraph relied on guava-12 and > DiskBackedPartitionStore used hasInt() - a method which does not exist > in guava-11 which hadoop 2 depends on. At runtime guava 11 was being > used > > I suppose this problem might indicate I'm running submitting the job > using the wrong binary. Currently I am including the giraph > dependencies with the jar, and running using hadoop jar. > > On 9/7/13, Claudio Martella wrote: > > OOC is used also at input superstep. try to decrease the number of > > partitions kept in memory. > > > > > > On Sat, Sep 7, 2013 at 1:37 AM, Alexander Asplund > > wrote: > > > >> Hi, > >> > >> I'm trying to process a graph that is about 3 times the size of > >> available memory. On the other hand, there is plenty of disk space. I > >> have enabled the giraph.useOutOfCoreGraph property, but it still > >> crashes with outOfMemoryError: GC limit exceeded when I try running my > >> job. > >> > >> I'm wondering of the spilling is supposed to work during the input > >> step. If so, are there any additional steps that must be taken to > >> ensure it functions? > >> > >> Regards, > >> Alexander Asplund > >> > > > > > > > > -- > > Claudio Martella > > claudio.martella@gmail.com > > > > > -- > Alexander Asplund > -- Claudio Martella claudio.martella@gmail.com --20cf30780c5c028f7404e5fba87d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
did you extend the heap available to the mapper tasks? e.g= . through mapred.child.java.opts.


<= div class=3D"gmail_quote">On Tue, Sep 10, 2013 at 12:50 AM, Alexander Asplu= nd <alexasplund@gmail.com> wrote:
Thanks for the reply.

I tried setting giraph.maxPartitionsInMemory to 1, but I'm still
getting OOM: GC limit exceeded.

Are there any particular cases the OOC will not be able to handle, or
is it supposed to work in all cases? If the latter, it might be that I
have made some configuration error.

I do have one concern that might indicateI have done something wrong:
to allow OOC to activate without crashing I had to modify the trunk
code. This was because Giraph relied on guava-12 and
DiskBackedPartitionStore used hasInt() - a method which does not exist
in guava-11 which hadoop 2 depends on. At runtime guava 11 was being
used

I suppose this problem might indicate I'm running submitting the job using the wrong binary. Currently I am including the giraph
dependencies with the jar, and running using hadoop jar.

On 9/7/13, Claudio Martella <claudio.martella@gmail.com> wrote:
> OOC is used also at input superstep. try to decrease the number of
> partitions kept in memory.
>
>
> On Sat, Sep 7, 2013 at 1:37 AM, Alexander Asplund
> <alexasplund@gmail.com= >wrote:
>
>> Hi,
>>
>> I'm trying to process a graph that is about 3 times the size o= f
>> available memory. On the other hand, there is plenty of disk space= . I
>> have enabled the giraph.useOutOfCoreGraph property, but it still >> crashes with outOfMemoryError: GC limit exceeded when I try runnin= g my
>> job.
>>
>> I'm wondering of the spilling is supposed to work during the i= nput
>> step. If so, are there any additional steps that must be taken to<= br> >> ensure it functions?
>>
>> Regards,
>> Alexander Asplund
>>
>
>
>
> --
> =A0 =A0Claudio Martella
> =A0 =A0claudio.martella@= gmail.com
>


--
Alexander Asplund



-- =A0 =A0Claudio Martella
=A0 =A0claudio.martella@gmail.com=A0 =A0 --20cf30780c5c028f7404e5fba87d--