Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@giraph.apache.org
Received-SPF: pass (nike.apache.org: domain of claudio.martella@gmail.com
 designates 209.85.216.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <52FD3026.3060403@apache.org>
References: 
 <CA+ULb+u-Mz=jW0A8jfZno4OHUcJszViyo+Fk-yEr34PuHbY5sg@mail.gmail.com>
 <52FAB4BE.2000005@apache.org>
 <CAFJOoJcQVjNhQtHO37JLE6FjWyewPhT9fcwfpWFBjsK5o+gQ3Q@mail.gmail.com>
 <52FB38C4.5050002@apache.org>
 <CAFJOoJcc7dzz0=oeVJthXW=-2M821+Ti04gYNNZjtJdEdQM7HA@mail.gmail.com>
 <20140212115025.GA692@imap.vu.nl> <20140212115318.GB692@imap.vu.nl>
 <52FB6266.9030202@apache.org>
 <CAFJOoJc-ock=fiyS8CxmPz+sS4s1UyDfb=rxe3zsZ8UGSp8fOw@mail.gmail.com>
 <52FB7E38.80102@apache.org> <20140212152125.GA686@imap.vu.nl>
 <52FD3026.3060403@apache.org>
From: Claudio Martella <claudio.martella@gmail.com>
Date: Sat, 15 Feb 2014 20:32:12 +0100
Message-ID: 
 <CAFJOoJeJSZcbPLBZ+Xf+YgONegvpF-Wn2KLd8TPn-j+=CspnJA@mail.gmail.com>
Subject: Re: GIRAPH-825 and GIRAPH-840
To: "dev@giraph.apache.org" <dev@giraph.apache.org>,
 Sebastian Schelter <ssc@apache.org>
Cc: Armando Miraglia <a.miraglia@student.vu.nl>
Content-Type: multipart/alternative; boundary=001a113a9cc4d9b77a04f276fb09

--001a113a9cc4d9b77a04f276fb09
Content-Type: text/plain; charset=ISO-8859-1

Sebastian, I had a look at your vertexinputformat. I think there might be a
bug. Why are you caching/reusing the id? This way every vertex parsed by
the vertexreader will share the same ID object, and hence have the same ID.
I think this is broken. you should instantiate a new ID object in the
preprocessLine.
Can you try like that?


On Thu, Feb 13, 2014 at 9:50 PM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi Armando,
>
> I uploaded my test code to github at:
>
> https://github.com/sscdotopen/giraph/tree/hyperball64-ooc
>
> I'm working on an algorithm to estimate the neighborhood function of the
> graph (similar to [1]). I'm running this on the transposed adjacency matrix
> of a snapshot of the twitter follower graph [2]. For this graph out-of-core
> is not necessary, but I would like to run my algorithm on another larger
> graph that doesn't fit into the aggregated main memory of the cluster
> anymore.
>
> I think for testing purposes, you can run it on any large graph in
> adjacency form.
>
> Our cluster consists of 25 machines with 32GB ram, 8 cores and 4 disks per
> machine. I use the following options to run the algorithm:
>
> hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
>
> org.apache.giraph.examples.hyperball.HyperBall
>
> --vertexInputFormat org.apache.giraph.examples.hyperball.
> HyperBallTextInputFormat
>
> --vertexInputPath hdfs:///ssc/twitter-negative/
>
> --vertexOutputFormat org.apache.giraph.io.formats.
> IdWithValueTextOutputFormat
>
> --outputPath hdfs:///ssc/tmp-123/
>
> --combiner org.apache.giraph.comm.messages.HyperLogLogCombiner
>
> --outEdges org.apache.giraph.edge.LongNullArrayEdges
>
> --workers 24
>
> --customArguments
>
> giraph.oneToAllMsgSending=true,
> giraph.isStaticGraph=true,
> giraph.numComputeThreads=15,
> giraph.numInputThreads=15,
> giraph.numOutputThreads=15,
> giraph.maxNumberOfSupersteps=30,
> giraph.useOutOfCoreGraph=true,
> giraph.maxPartitionsInMemory=20
>
> Best,
> Sebastian
>
> [1] http://arxiv.org/abs/1308.2144
> [2] http://konect.uni-koblenz.de/networks/twitter_mpi
>
>
> On 02/12/2014 04:21 PM, Armando Miraglia wrote:
>
>>
>> Hi Sebastian,
>>
>> On Wed, Feb 12, 2014 at 02:59:20PM +0100, Sebastian Schelter wrote:
>>
>>> No. Should I have done that?
>>>
>>
>> could you please provide me with the test you have done together with
>> the variables that you have set during for the computation? This would
>> help me a lot.
>>
>> Cheers,
>> Armando
>>
>>
>


-- 
   Claudio Martella

--001a113a9cc4d9b77a04f276fb09--