Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9D9911813A for ; Mon, 14 Mar 2016 16:36:07 +0000 (UTC) Received: (qmail 65345 invoked by uid 500); 14 Mar 2016 16:36:07 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 65256 invoked by uid 500); 14 Mar 2016 16:36:07 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 65246 invoked by uid 99); 14 Mar 2016 16:36:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2016 16:36:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id CE66B1A0257 for ; Mon, 14 Mar 2016 16:36:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.73 X-Spam-Level: X-Spam-Status: No, score=-2.73 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.329, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=mailbox.org Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id mYELEnt1fuX1 for ; Mon, 14 Mar 2016 16:36:04 +0000 (UTC) Received: from mx2.mailbox.org (mx2.mailbox.org [80.241.60.215]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id AAA6A5FB57 for ; Mon, 14 Mar 2016 16:36:03 +0000 (UTC) Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx2.mailbox.org (Postfix) with ESMTPS id EF37442105 for ; Mon, 14 Mar 2016 17:35:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mailbox.org; h= content-transfer-encoding:content-type:content-type:in-reply-to :mime-version:date:date:message-id:from:from:references:subject :subject:received; s=mail20150812; t=1457973355; bh=b4xEr8u7tc4t cEoYbGQEmXrtoCDs+tx9xGGEOR0oP+w=; b=oWNEulHM//i5RibIRXY/uvIOIgyl C0X9XxrSFKaFVMwDyrrf1H22LQbooillYfjhJSTegNAXWLNIUNINyxVka8dE9874 29fhhrCIjGuTSWQad5udVpXoVF+MM4rRr3Lxr9ifQULIq3v8OEfWffYE42+GD26a lUbgddnQVSi5nif2GgJHwzRALOcyf5qjd6r2qh0piuJtpKdW4X+i+rxRTsZW/zy7 z+ct66WYoZrOgJ7OCmeeaaJI7a69CQVYrrFHmY0mQIbtlEWBpKidcUUlDkUX95Xj OG/N/4BqUX+eHdXh/ewJC8XFHBH/Z+HD8pij2iZP9XXsO2OVofnvAfNjJg== X-Virus-Scanned: amavisd-new at heinlein-support.de Received: from smtp1.mailbox.org ([80.241.60.240]) by gerste.heinlein-support.de (gerste.heinlein-support.de [91.198.250.173]) (amavisd-new, port 10030) with ESMTP id HKpdt6yC0Wvf for ; Mon, 14 Mar 2016 17:35:55 +0100 (CET) Subject: Re: Memory ran out PageRank To: user@flink.apache.org References: <6004B89B-A925-488D-8685-FA2D81D76753@inria.fr> From: Martin Junghanns Message-ID: <56E6E88A.6070300@mailbox.org> Date: Mon, 14 Mar 2016 17:36:26 +0100 MIME-Version: 1.0 In-Reply-To: <6004B89B-A925-488D-8685-FA2D81D76753@inria.fr> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Hi I think this is the same issue we had before on the list [1]. Stephan recommended the following workaround: > A possible workaround is to use the option "setSolutionSetUnmanaged(true)" > on the iteration. That will eliminate the fragmentation issue, at least. Unfortunately, you cannot set this when using graph.run(new PageRank(...)) I created a Gist which shows you how to set this using PageRank https://gist.github.com/s1ck/801a8ef97ce374b358df Please let us know if it worked out for you. Cheers, Martin [1] http://mail-archives.apache.org/mod_mbox/flink-user/201508.mbox/%3CCAELUF_ByPAB%2BPXWLemPzRH%3D-awATeSz4sGz4v9TmnvFku3%3Dx3A%40mail.gmail.com%3E On 14.03.2016 16:55, Ovidiu-Cristian MARCU wrote: > Hi, > > While running PageRank on a synthetic graph I run into this problem: > Any advice on how should I proceed to overcome this memory issue? > > IterationHead(Vertex-centric iteration (org.apache.flink.graph.library.PageRank$VertexRankUpdater@7712cae0 | org.apache.flink.graph.library.PageRank$RankMesseng$ > java.lang.RuntimeException: Memory ran out. Compaction failed. numPartitions: 32 minPartition: 24 maxPartition: 25 number of overflow segments: 328 bucketSize: 638 Overall memory: 115539968 Partition memory: 50659328 Message: Index: 25, Size: 24 > at org.apache.flink.runtime.operators.hash.CompactingHashTable.insertRecordIntoPartition(CompactingHashTable.java:469) > at org.apache.flink.runtime.operators.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:414) > at org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTableWithUniqueKey(CompactingHashTable.java:325) > at org.apache.flink.runtime.iterative.task.IterationHeadTask.readInitialSolutionSet(IterationHeadTask.java:212) > at org.apache.flink.runtime.iterative.task.IterationHeadTask.run(IterationHeadTask.java:273) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) > at java.lang.Thread.run(Thread.java:745) > > Thanks! > > Best, > Ovidiu >