Return-Path: 
 <dev-return-9320-apmail-giraph-dev-archive=giraph.apache.org@giraph.apache.org>
X-Original-To: apmail-giraph-dev-archive@www.apache.org
Delivered-To: apmail-giraph-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8B9F81083B
	for <apmail-giraph-dev-archive@www.apache.org>;
 Thu, 13 Feb 2014 20:51:16 +0000 (UTC)
Received: (qmail 43134 invoked by uid 500); 13 Feb 2014 20:51:15 -0000
Delivered-To: apmail-giraph-dev-archive@giraph.apache.org
Received: (qmail 43083 invoked by uid 500); 13 Feb 2014 20:51:15 -0000
Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@giraph.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@giraph.apache.org>
List-Post: <mailto:dev@giraph.apache.org>
List-Id: <dev.giraph.apache.org>
Reply-To: dev@giraph.apache.org
Delivered-To: mailing list dev@giraph.apache.org
Received: (qmail 43075 invoked by uid 99); 13 Feb 2014 20:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Feb 2014 20:51:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ssc.open@googlemail.com
 designates 209.85.214.49 as permitted sender)
Received: from [209.85.214.49] (HELO mail-bk0-f49.google.com) (209.85.214.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Feb 2014 20:51:09 +0000
Received: by mail-bk0-f49.google.com with SMTP id v15so3215041bkz.22
        for <dev@giraph.apache.org>; Thu, 13 Feb 2014 12:50:49 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=message-id:date:from:reply-to:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        bh=ajY2kdk9WqqytZD2fpasLgWfF3OXgQJ4h5WNDpkzoxM=;
        b=XZxsT3wgZtpCq98ajUTRvhzFStt+VKeVzMqLrksBs5TSH9QMYk7JZ8EKoXEHup7wg6
         Wd12yHBFU9WC1us7HhNCar6omIO1PHZIiFuGtaWvlJxMVZ945e0A2BUdt8tvTd01+E4D
         T3Zt9ApgWIkUbrVQyL2KALbFL9jS/jxdCX8WXID9demKLm0Wx06LlLmFpIA79js5ztP0
         F/BkIxDFnVS4Lzz0dzfNzadVH9mGpixpw5A0UU+KtZ8wuCf+qt/xkrkTpXA7th5OLs9x
         vHc++K6bc/E22vwtLn8tC7059d3h6TIPHSngpWNSxwye9Dj3dAM5GoFTsHi6X0a0677Y
         v6dw==
X-Received: by 10.205.34.140 with SMTP id ss12mr42188bkb.121.1392324649415;
        Thu, 13 Feb 2014 12:50:49 -0800 (PST)
Received: from [192.168.0.2] (g229126112.adsl.alicedsl.de. [92.229.126.112])
        by mx.google.com with ESMTPSA id
 tf11sm2972305bkb.17.2014.02.13.12.50.48
        for <multiple recipients>
        (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Thu, 13 Feb 2014 12:50:48 -0800 (PST)
Message-ID: <52FD3026.3060403@apache.org>
Date: Thu, 13 Feb 2014 21:50:46 +0100
From: Sebastian Schelter <ssc@apache.org>
Reply-To: ssc@apache.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.2.0
MIME-Version: 1.0
To: Armando Miraglia <a.miraglia@student.vu.nl>, dev@giraph.apache.org
Subject: Re: GIRAPH-825 and GIRAPH-840
References: 
 <CA+ULb+u-Mz=jW0A8jfZno4OHUcJszViyo+Fk-yEr34PuHbY5sg@mail.gmail.com>
 <52FAB4BE.2000005@apache.org>
 <CAFJOoJcQVjNhQtHO37JLE6FjWyewPhT9fcwfpWFBjsK5o+gQ3Q@mail.gmail.com>
 <52FB38C4.5050002@apache.org>
 <CAFJOoJcc7dzz0=oeVJthXW=-2M821+Ti04gYNNZjtJdEdQM7HA@mail.gmail.com>
 <20140212115025.GA692@imap.vu.nl> <20140212115318.GB692@imap.vu.nl>
 <52FB6266.9030202@apache.org>
 <CAFJOoJc-ock=fiyS8CxmPz+sS4s1UyDfb=rxe3zsZ8UGSp8fOw@mail.gmail.com>
 <52FB7E38.80102@apache.org> <20140212152125.GA686@imap.vu.nl>
In-Reply-To: <20140212152125.GA686@imap.vu.nl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Armando,

I uploaded my test code to github at:

https://github.com/sscdotopen/giraph/tree/hyperball64-ooc

I'm working on an algorithm to estimate the neighborhood function of the 
graph (similar to [1]). I'm running this on the transposed adjacency 
matrix of a snapshot of the twitter follower graph [2]. For this graph 
out-of-core is not necessary, but I would like to run my algorithm on 
another larger graph that doesn't fit into the aggregated main memory of 
the cluster anymore.

I think for testing purposes, you can run it on any large graph in 
adjacency form.

Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks 
per machine. I use the following options to run the algorithm:

hadoop jar 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner 


org.apache.giraph.examples.hyperball.HyperBall

--vertexInputFormat 
org.apache.giraph.examples.hyperball.HyperBallTextInputFormat

--vertexInputPath hdfs:///ssc/twitter-negative/

--vertexOutputFormat 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat

--outputPath hdfs:///ssc/tmp-123/

--combiner org.apache.giraph.comm.messages.HyperLogLogCombiner

--outEdges org.apache.giraph.edge.LongNullArrayEdges

--workers 24

--customArguments

giraph.oneToAllMsgSending=true,
giraph.isStaticGraph=true,
giraph.numComputeThreads=15,
giraph.numInputThreads=15,
giraph.numOutputThreads=15,
giraph.maxNumberOfSupersteps=30,
giraph.useOutOfCoreGraph=true,
giraph.maxPartitionsInMemory=20

Best,
Sebastian

[1] http://arxiv.org/abs/1308.2144
[2] http://konect.uni-koblenz.de/networks/twitter_mpi

On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>
> Hi Sebastian,
>
> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>> No. Should I have done that?
>
> could you please provide me with the test you have done together with
> the variables that you have set during for the computation? This would
> help me a lot.
>
> Cheers,
> Armando
>