Return-Path: X-Original-To: apmail-hama-user-archive@www.apache.org Delivered-To: apmail-hama-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 93200DC6F for ; Fri, 14 Sep 2012 07:28:45 +0000 (UTC) Received: (qmail 93968 invoked by uid 500); 14 Sep 2012 07:28:45 -0000 Delivered-To: apmail-hama-user-archive@hama.apache.org Received: (qmail 93875 invoked by uid 500); 14 Sep 2012 07:28:44 -0000 Mailing-List: contact user-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hama.apache.org Delivered-To: mailing list user@hama.apache.org Received: (qmail 93850 invoked by uid 99); 14 Sep 2012 07:28:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 07:28:44 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of thomas.jungblut@gmail.com designates 209.85.220.175 as permitted sender) Received: from [209.85.220.175] (HELO mail-vc0-f175.google.com) (209.85.220.175) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 07:28:35 +0000 Received: by vcdm8 with SMTP id m8so4864316vcd.34 for ; Fri, 14 Sep 2012 00:28:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=6lg1M5Xsa0oHoYqGiPiGzAgJ2txKwbR/RbejEw2jglw=; b=Oj8XPapBJAko2VjwoYwOa/kveROsFUP2GNW1BH2gkfhOqUYuzC5ad7kEGAtLkwhHE2 YpT3yWkt0FixIfOXW56rHiE0UEic+2Rsbi8JIaupkSPv5n3ZvgH6nUJFO6xeWysT439h tYnSF88lw9V1DvUyMbMB3pIya0qU1vxufdt1gVPShQWyCWe/nzNMuvoZbT6RWfk3Z1wz WnMH3Az0iuAqkHhQPkxZcakswEQQT9L22Ua57uaMCqNs+jd7HPtWj4AT+h4wGKryDhL/ YasOVW3BJmd8r5slNqu+UV2lJR0a5mepX8feqPsuTSbCr0mX4THmupVcFoEC/+P01O1D rLKA== MIME-Version: 1.0 Received: by 10.58.58.161 with SMTP id s1mr1365413veq.14.1347607694934; Fri, 14 Sep 2012 00:28:14 -0700 (PDT) Received: by 10.220.108.3 with HTTP; Fri, 14 Sep 2012 00:28:14 -0700 (PDT) In-Reply-To: References: Date: Fri, 14 Sep 2012 09:28:14 +0200 Message-ID: Subject: Re: out of memory problem... From: Thomas Jungblut To: user@hama.apache.org Content-Type: multipart/alternative; boundary=047d7b66f86ff4047f04c9a45d29 --047d7b66f86ff4047f04c9a45d29 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable I have created an issue for that: HAMA-642 2012/9/14 Thomas Jungblut > Basically I think that the graph should fit into memory of your task. > So the messages could cause the overflow. > > You can try out the DiskQueue, this can be configured with setting the > property "hama.messenger.queue.class" to > "org.apache.hama.bsp.message.DiskQueue". > > This will immediately flush the messages to disk. However this is > experimental currently, so if you try it out please tell us if it helped. > > Thanks. > > To further scale this, we should write vertices that don't fit in memory > on the disk. I will add another jira for that soon. > > 2012/9/14 =D7=AF=BF=CB=E8=A1 > >> oh, the HDFS block size is 128Mb, not 64Mb, so the 73Mb graph will not >> be split-ed on the HDFS. >> >> 2012/9/14 =D7=AF=BF=CB=E8=A1 >> >> > em... I have try your configure advise and restart the hama. >> > I use the Google web graph( >> > http://wiki.apache.org/hama/WriteHamaGraphFile ), >> > Nodes: 875713 Edges: 5105039, which is about 73Mb, upload to a small >> HDFS >> > cluster(block size is 64Mb), test the PageRank in ( >> > http://wiki.apache.org/hama/WriteHamaGraphFile ), got the result as: >> > ################ >> > function@624-PC:~/hadoop-1.0.3/hama-0.6.0$ hama jar hama-6-P* >> > input-google ouput-google >> > 12/09/14 14:27:50 INFO bsp.FileInputFormat: Total input paths to >> process : >> > 1 >> > 12/09/14 14:27:50 INFO bsp.FileInputFormat: Total # of splits: 3 >> > 12/09/14 14:27:50 INFO bsp.BSPJobClient: Running job: >> job_201008141420_0004 >> > 12/09/14 14:27:53 INFO bsp.BSPJobClient: Current supersteps number: 0 >> > Java HotSpot(TM) Server VM warning: Attempt to allocate stack guard >> pages >> > failed. >> > ################### >> > >> > Last time the supersteps could be 1 or 2, then the same result. >> > the task attempt****.err files are empty. >> > Is the graph too large? >> > I test on a small graph, get the right Rank results >> > >> > >> > 2012/9/14 Edward J. Yoon >> > >> > I've added multi-step partitioning method to save memory[1]. >> >> >> >> Please try to configure below property to hama-site.xml. >> >> >> >> >> >> hama.graph.multi.step.partitioning.interval >> >> 10000000 >> >> >> >> >> >> 1. https://issues.apache.org/jira/browse/HAMA-599 >> >> >> >> On Fri, Sep 14, 2012 at 3:13 PM, =D7=AF=BF=CB=E8=A1 wrote: >> >> > HI, Actually I use this ( >> >> > >> >> >> https://builds.apache.org/job/Hama-Nightly/672/artifact/.repository/org/= apache/hama/hama-dist/0.6.0-SNAPSHOT/ >> >> > ) >> >> > to test again, I mean use this 0.6.0SNAPSHOT version replace >> everything, >> >> > got the same out of memory results. I just don't know what cause th= e >> >> out of >> >> > memory fails, only some small graph computing can be finished. Is >> this >> >> > version finished the " >> >> > [HAMA-596]:Optimize >> >> > memory usage of graph job" ? >> >> > Thanks >> >> > >> >> > 2012/9/14 Thomas Jungblut >> >> > >> >> >> Hey, what jar did you exactly replace? >> >> >> Am 14.09.2012 07:49 schrieb "=D7=AF=BF=CB=E8=A1" : >> >> >> >> >> >> > hi, every one: >> >> >> > I use the hama-0.5.0 with the hadoop-1.0.3, try to do some large >> >> graphs >> >> >> > analysis. >> >> >> > When I test the PageRank examples, as the ( >> >> >> > http://wiki.apache.org/hama/WriteHamaGraphFile) shows, I downloa= d >> >> the >> >> >> > graph >> >> >> > data, and run the PageRank job on a small distributed cluser, I >> can >> >> only >> >> >> > get the out of memory failed, with Superstep 0,1,2 works well, >> then >> >> get >> >> >> the >> >> >> > memory out fail.(Each computer have 2G memory) But when I test >> some >> >> small >> >> >> > graph, everything went well. >> >> >> > Also I try the trunk version( >> >> >> > https://builds.apache.org/job/Hama-Nightly/672/changes#detail3), >> >> replace >> >> >> > my >> >> >> > hama-0.5.0 with the hama-0.6.0-snapshot, only get the same >> results. >> >> >> > Anyone got better ideas? >> >> >> > >> >> >> > Thanks! >> >> >> > >> >> >> > -- >> >> >> > >> >> >> > *Zhuang Kechen >> >> >> > * >> >> >> > >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > >> >> > *Zhuang Kechen* >> >> > >> >> > School of Computer Science & Technology >> >> > >> >> > ** >> >> > Nanjing University of Science & Technology >> >> > >> >> > Lab.623, School of Computer Sci. & Tech. >> >> > >> >> > No.200, Xiaolingwei Street >> >> > >> >> > Nanjing, Jiangsu, 210094 >> >> > >> >> > P.R. China >> >> > >> >> > Tel: 025-84315982** >> >> > >> >> > Email: zhuangkechen@gmail.com >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> >> >> > >> > >> > >> > -- >> > >> > *Zhuang Kechen >> > * >> > >> > >> > >> >> >> -- >> >> *Zhuang Kechen* >> >> School of Computer Science & Technology >> >> ** >> Nanjing University of Science & Technology >> >> Lab.623, School of Computer Sci. & Tech. >> >> No.200, Xiaolingwei Street >> >> Nanjing, Jiangsu, 210094 >> >> P.R. China >> >> Tel: 025-84315982** >> >> Email: zhuangkechen@gmail.com >> > > --047d7b66f86ff4047f04c9a45d29--