Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C190106CF for ; Tue, 13 Aug 2013 23:28:29 +0000 (UTC) Received: (qmail 63379 invoked by uid 500); 13 Aug 2013 23:28:29 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 63347 invoked by uid 500); 13 Aug 2013 23:28:29 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 63339 invoked by uid 99); 13 Aug 2013 23:28:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 23:28:29 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of prvs=1937b71a3f=majakabiljo@fb.com designates 67.231.153.30 as permitted sender) Received: from [67.231.153.30] (HELO mx0a-00082601.pphosted.com) (67.231.153.30) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Aug 2013 23:28:24 +0000 Received: from pps.filterd (m0004003 [127.0.0.1]) by mx0b-00082601.pphosted.com (8.14.5/8.14.5) with SMTP id r7DNOZFu025122 for ; Tue, 13 Aug 2013 16:28:02 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=from : to : subject : date : message-id : in-reply-to : content-type : mime-version; s=facebook; bh=vtCUhDhMKeSxxL92bs1ZIW14PyqdHET0laodcfDLm4Q=; b=an/Ns7zCt5gG0U6mlWciHlZB+It6fttPYa6n5Yv0IsR1gzx/3+Nff98AUVcSfFm9nFkn I72JxmD23EGeNAf+CW2Tv0DQvItRIV5yvLdjOUVQQK4hGmwxDSWm4o9Sz9UUPsqbbJXz l3Zovu2TK3YX7TfXbh7TSCilKRN7jDGWUzU= Received: from mail.thefacebook.com (prn1-cmdf-dc01-fw1-nat.corp.tfbnw.net [173.252.71.129] (may be forged)) by mx0b-00082601.pphosted.com with ESMTP id 1e7veh8fag-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK) for ; Tue, 13 Aug 2013 16:28:02 -0700 Received: from PRN-MBX02-2.TheFacebook.com ([169.254.5.142]) by PRN-CHUB04.TheFacebook.com ([fe80::7ded:c10e:ef04:80d8%12]) with mapi id 14.03.0146.000; Tue, 13 Aug 2013 16:28:00 -0700 From: Maja Kabiljo To: "user@giraph.apache.org" Subject: Re: Problems with sending/receiving messages of type IntArrayListWritable Thread-Topic: Problems with sending/receiving messages of type IntArrayListWritable Thread-Index: AQHOlTOCqwMGXSoge0KgwdgtiRyylJmOKgOAgAO+9ICAADOhAIAB4AeAgAAp7ACAAAK9AIAACvqA//+a1AA= Date: Tue, 13 Aug 2013 23:27:59 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.16.4] Content-Type: multipart/alternative; boundary="_000_CE300E0EBE3Cmajakabiljofbcom_" MIME-Version: 1.0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794,1.0.431,0.0.0000 definitions=2013-08-13_09:2013-08-13,2013-08-13,1970-01-01 signatures=0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE300E0EBE3Cmajakabiljofbcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable There is a bug in ArrayListWritable =96 readFileds should clear the list be= fore reading any data to it. We are reusing message objects and every time = we call readFields data is just going to be appended to the previous messag= e. It's very easy to fix, Kyle please feel free to open an issue and submit= a patch for it. From: Kyle Orlando > Reply-To: "user@giraph.apache.org" > Date: Tuesday, August 13, 2013 3:30 PM To: "user@giraph.apache.org" > Subject: Re: Problems with sending/receiving messages of type IntArrayListW= ritable Tried it, got the same exact results. Do you think it's a problem with Arr= ayListWritable, or with sendMessage()? I don't have a great understanding = of how sendMessage() utilizes the Writable interface exactly... I'm guessin= g that at some point it uses readFields() and write() because those obvious= ly need to be implemented for a class to be considered "Writable". Looking= at those two methods for ArrayListWritable, nothing jumps out at me as bei= ng a bug, though I can see where the minimum number of values in the ArrayL= ist could be set incorrectly in readFields(): int numValues =3D in.readInt(); // read number of values ensureCapacity(numValues); What do you think? On Tue, Aug 13, 2013 at 5:50 PM, Claudio Martella > wrote: could you try using: -D giraph.oneToAllMsgSending=3Dtrue when you run it? On Tue, Aug 13, 2013 at 11:41 PM, Kyle Orlando > wrote: Yep, and I've pulled recently as well. On Tue, Aug 13, 2013 at 3:10 PM, Claudio Martella > wrote: That's strange. This looks like a bug to me. Are you using trunk? On Mon, Aug 12, 2013 at 4:32 PM, Kyle Orlando > wrote: Also, I realize that I just sent an uncommented version... sorry guys, but the algorithm consists of just three supersteps: Superstep 0.) Source sends list of source's neighbors to each neighbor Superstep 1.) Neighbor receives list of source's neighbors from source, counts number of mutual neighbors, sends this value (in a 1-element list) back to source Superstep 2.) Source receives messages from neighbors that contain number of mutual neighbors. These are summed up and divided by the number of possible connections between neighbors to get the local clustering coefficient (between 0 and 1). This is set as the source vertex's value. On Mon, Aug 12, 2013 at 7:28 AM, Kyle Orlando > wrote: > Alrighty, here's my code: > > public void compute( > Vertex v= ertex, > Iterable messages) throws IOE= xception { > > if (isSource(vertex)) { > if (getSuperstep() =3D=3D 0) { > System.out.println("\nSUPERSTEP 0\n----= --------------------------"); > IntArrayListWritable sourceNeighbors = =3D new IntArrayListWritable(); > > vertex.setValue(new DoubleWritable(0)); > > for (Edge ed= ge : vertex.getEdges()) { > IntWritable targetVertex =3D ne= w > IntWritable(edge.getTargetVertexId().get()); > sourceNeighbors.add(targetVerte= x); > } > > for (IntWritable neighbor : sourceNeigh= bors) { > sendMessage(neighbor, sourceNei= ghbors); > System.out.println("Vertex " + = vertex.getId() + " sends " + > sourceNeighbors + " to " + neighbor); > } > > } else if (getSuperstep() =3D=3D 2) { > int total =3D vertex.getNumEdges(); > int possibleCombos =3D total * (total -= 1); > int mutualNeighbors =3D 0; > double coefficient =3D 0; > int count =3D 1; > > System.out.println("\nSUPERSTEP 2\n----= --------------------------"); > > for (IntArrayListWritable message : mes= sages) { > System.out.println("Message " += count + " contains: " + message); > mutualNeighbors +=3D message.ge= t(0).get(); > count++; > } > > if (possibleCombos > 0) { > coefficient =3D (double)mutualN= eighbors / possibleCombos; > vertex.setValue(new DoubleWrita= ble(coefficient)); > } > > } > } else { > vertex.setValue(new DoubleWritable(0)); > > if (getSuperstep() =3D=3D 1) { > int mutualNeighbors =3D 0; > > System.out.println("\nSUPERSTEP 1\n----= --------------------------"); > IntArrayListWritable sourceNeighbors = =3D messages.iterator().next(); > System.out.println("Vertex " + vertex.g= etId() + " receives " + > sourceNeighbors); > > for (Edge ed= ge : vertex.getEdges()) { > IntWritable targetVertex =3D ne= w > IntWritable(edge.getTargetVertexId().get()); > if (sourceNeighbors.contains(ta= rgetVertex)) mutualNeighbors++; > } > > IntArrayListWritable neighborCount =3D = new IntArrayListWritable(); > neighborCount.add(new IntWritable(mutua= lNeighbors)); > > sendMessage(new IntWritable(SOURCE_ID.g= et(getConf())), neighborCount); > System.out.println("Vertex " + vertex.g= etId() + " sends " + > neighborCount + " to Vertex " + SOURCE_ID.get(getConf())); > > } > } > > vertex.voteToHalt(); > } > > > On Fri, Aug 9, 2013 at 10:15 PM, Claudio Martella > > wrote: >> you'd have to show us the code in the compute method. your problem might= be >> caused by object reusal. >> >> >> On Fri, Aug 9, 2013 at 9:05 PM, Kyle Orlando > >> wrote: >>> >>> Hello, >>> >>> I am trying to write code to compute the local clustering coefficient >>> of a vertex/some vertices, and to do this I send a message that >>> contains a list of the source's neighbors to each of its neighbors. >>> This is in, of course, an IntArrayListWritable. I check the list that >>> I am sending before invoking sendMessage(), and it appears to be >>> correct. However, when I use message.iterator.next() or something >>> similar, the IntArrayListWritable objects seem to repeat or something. >>> I did some logging, and here was the output: >>> >>> SUPERSTEP 0 >>> ------------------------------ >>> Vertex 1 sends [2, 3, 4] to Vertex 2 >>> Vertex 1 sends [2, 3, 4] to Vertex 3 >>> Vertex 1 sends [2, 3, 4] to Vertex 4 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 2 receives [2, 3, 4] >>> Vertex 2 sends [1] to Vertex 1 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 3 receives [2, 3, 4, 2, 3, 4] >>> Vertex 3 sends [1] to Vertex 1 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 4 receives [2, 3, 4, 2, 3, 4, 2, 3, 4] >>> Vertex 4 sends [0] to Vertex 1 >>> >>> SUPERSTEP 2 >>> ------------------------------ >>> Message 1 contains: [1] >>> Message 2 contains: [1, 1, 1] >>> Message 3 contains: [1, 1, 1, 1, 1, 0] >>> >>> What is happening? >>> >>> -- >>> Kyle Orlando >>> Computer Engineering Major >>> University of Maryland >> >> >> >> >> -- >> Claudio Martella >> claudio.martella@gmail.com > > > > -- > Kyle Orlando > Computer Engineering Major > University of Maryland -- Kyle Orlando Computer Engineering Major University of Maryland -- Claudio Martella claudio.martella@gmail.com -- Kyle Orlando Computer Engineering Major University of Maryland -- Claudio Martella claudio.martella@gmail.com -- Kyle Orlando Computer Engineering Major University of Maryland --_000_CE300E0EBE3Cmajakabiljofbcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <8AB7981DC3E5D24CAD279D2A9FCBEB04@fb.com> Content-Transfer-Encoding: quoted-printable
There is a bug in ArrayListWritable =96 readFileds should clear the li= st before reading any data to it. We are reusing message objects and every = time we call readFields data is just going to be appended to the previous m= essage. It's very easy to fix, Kyle please feel free to open an issue and submit a patch for it.

From: Kyle Orlando <kyle.r.orlando@gmail.com>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Tuesday, August 13, 2013 3:30= PM
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Problems with sending/= receiving messages of type IntArrayListWritable

Tried it, got the same exact results.  Do you think it's a proble= m with ArrayListWritable, or with sendMessage()?  I don't have a great= understanding of how sendMessage() utilizes the Writable interface exactly= ... I'm guessing that at some point it uses readFields() and write() because those obviously need to be implemented fo= r a class to be considered "Writable".  Looking at those two= methods for ArrayListWritable, nothing jumps out at me as being a bug, tho= ugh I can see where the minimum number of values in the ArrayList could be set incorrectly in readFields():

    int numValues =3D in.readInt();    &= nbsp;       // read number of values
    ensureCapacity(numValues);

What do you think?


On Tue, Aug 13, 2013 at 5:50 PM, Claudio Martell= a <claudio= .martella@gmail.com> wrote:
could you try using:

-D g= iraph.oneToAllMsgSending=3Dtrue when you run it?


On Tue, Aug 13, 2013 at 11:41 PM, Kyle Orlando <= span dir=3D"ltr"> <kyle.r.or= lando@gmail.com> wrote:
Yep, and I've pulled recently as well.


On Tue, Aug 13, 2013 at 3:10 PM, Claudio Martell= a <claudio= .martella@gmail.com> wrote:
That's strange. This looks like a bug to me. Are you using= trunk?


On Mon, Aug 12, 2013 at 4:32 PM, Kyle Orlando <kyle.r.or= lando@gmail.com> wrote:
Also, I realize that I just sent an uncommented version... sorry guys,
but the algorithm consists of just three supersteps:


Superstep 0.) Source sends list of source's neighbors to each neighbor

Superstep 1.) Neighbor receives list of source's neighbors from
source, counts number of mutual neighbors, sends this value (in a
1-element list) back to source

Superstep 2.) Source receives messages from neighbors that contain
number of mutual neighbors. These are summed up and divided by the
number of possible connections between neighbors to get the local
clustering coefficient (between 0 and 1). This is set as the source
vertex's value.

On Mon, Aug 12, 2013 at 7:28 AM, Kyle Orlando <kyle.r.orlando@gmail.com> wrote= :
> Alrighty, here's my code:
>
>           public void compute(
>                     =   Vertex<IntWritable, DoubleWritable, NullWritable> vertex,
>                     =   Iterable<IntArrayListWritable> messages) throws IOException {<= br> >
>                   if (isS= ource(vertex)) {
>                     =       if (getSuperstep() =3D=3D 0) {
>                     =               System.out.println("\= nSUPERSTEP 0\n------------------------------");
>                     =               IntArrayListWritable sourc= eNeighbors =3D new IntArrayListWritable();
>
>                     =               vertex.setValue(new Double= Writable(0));
>
>                     =               for (Edge<IntWritable, = NullWritable> edge : vertex.getEdges()) {
>                     =                      = ; IntWritable targetVertex =3D new
> IntWritable(edge.getTargetVertexId().get());
>                     =                      = ; sourceNeighbors.add(targetVertex);
>                     =               }
>
>                     =               for (IntWritable neighbor = : sourceNeighbors) {
>                     =                      = ; sendMessage(neighbor, sourceNeighbors);
>                     =                      = ; System.out.println("Vertex " + vertex.getId() + " = sends " +
> sourceNeighbors + " to " + neighbor);
>                     =               }
>
>                     =       } else if (getSuperstep() =3D=3D 2) {
>                     =               int total =3D vertex.getNu= mEdges();
>                     =               int possibleCombos =3D tot= al * (total - 1);
>                     =               int mutualNeighbors =3D 0;=
>                     =               double coefficient =3D 0;<= br> >                     =               int count =3D 1;
>
>                     =               System.out.println("\= nSUPERSTEP 2\n------------------------------");
>
>                     =               for (IntArrayListWritable = message : messages) {
>                     =                      = ; System.out.println("Message " + count + " contains= : " + message);
>                     =                      = ; mutualNeighbors +=3D message.get(0).get();
>                     =                      = ; count++;
>                     =               }
>
>                     =               if (possibleCombos > 0)= {
>                     =                      = ; coefficient =3D (double)mutualNeighbors / possibleCombos;
>                     =                      = ; vertex.setValue(new DoubleWritable(coefficient));
>                     =               }
>
>                     =       }
>                   } else = {
>                     =       vertex.setValue(new DoubleWritable(0));
>
>                     =       if (getSuperstep() =3D=3D 1) {
>                     =               int mutualNeighbors =3D 0;=
>
>                     =               System.out.println("\= nSUPERSTEP 1\n------------------------------");
>                     =               IntArrayListWritable sourc= eNeighbors =3D messages.iterator().next();
>                     =               System.out.println("V= ertex " + vertex.getId() + " receives " +
> sourceNeighbors);
>
>                     =               for (Edge<IntWritable, = NullWritable> edge : vertex.getEdges()) {
>                     =                      = ; IntWritable targetVertex =3D new
> IntWritable(edge.getTargetVertexId().get());
>                     =                      = ; if (sourceNeighbors.contains(targetVertex)) mutualNeighbors++; >                     =               }
>
>                     =               IntArrayListWritable neigh= borCount =3D new IntArrayListWritable();
>                     =               neighborCount.add(new IntW= ritable(mutualNeighbors));
>
>                     =               sendMessage(new IntWritabl= e(SOURCE_ID.get(getConf())), neighborCount);
>                     =               System.out.println("V= ertex " + vertex.getId() + " sends " +
> neighborCount + " to Vertex " + SOURCE_ID.get(getCon= f()));
>
>                     =       }
>                   }
>
>                 vertex.voteToH= alt();
>           }
>
>
> On Fri, Aug 9, 2013 at 10:15 PM, Claudio Martella
> <cl= audio.martella@gmail.com> wrote:
>> you'd have to show us the code in the compute method. your problem= might be
>> caused by object reusal.
>>
>>
>> On Fri, Aug 9, 2013 at 9:05 PM, Kyle Orlando <kyle.r.orlando@gmail.com&g= t;
>> wrote:
>>>
>>> Hello,
>>>
>>> I am trying to write code to compute the local clustering coef= ficient
>>> of a vertex/some vertices, and to do this I send a message tha= t
>>> contains a list of the source's neighbors to each of its neigh= bors.
>>> This is in, of course, an IntArrayListWritable.  I check = the list that
>>> I am sending before invoking sendMessage(), and it appears to = be
>>> correct. However, when I use message.iterator.next() or someth= ing
>>> similar, the IntArrayListWritable objects seem to repeat or so= mething.
>>> I did some logging, and here was the output:
>>>
>>> SUPERSTEP 0
>>> ------------------------------
>>> Vertex 1 sends [2, 3, 4] to Vertex 2
>>> Vertex 1 sends [2, 3, 4] to Vertex 3
>>> Vertex 1 sends [2, 3, 4] to Vertex 4
>>>
>>> SUPERSTEP 1
>>> ------------------------------
>>> Vertex 2 receives [2, 3, 4]
>>> Vertex 2 sends [1] to Vertex 1
>>>
>>> SUPERSTEP 1
>>> ------------------------------
>>> Vertex 3 receives [2, 3, 4, 2, 3, 4]
>>> Vertex 3 sends [1] to Vertex 1
>>>
>>> SUPERSTEP 1
>>> ------------------------------
>>> Vertex 4 receives [2, 3, 4, 2, 3, 4, 2, 3, 4]
>>> Vertex 4 sends [0] to Vertex 1
>>>
>>> SUPERSTEP 2
>>> ------------------------------
>>> Message 1 contains: [1]
>>> Message 2 contains: [1, 1, 1]
>>> Message 3 contains: [1, 1, 1, 1, 1, 0]
>>>
>>> What is happening?
>>>
>>> --
>>> Kyle Orlando
>>> Computer Engineering Major
>>> University of Maryland
>>
>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland



--
Kyle Orlando
Computer Engineering Major
University of Maryland



--
   Claudio Martella
   claudio.martella@gmail.com   



--
Kyle Orlando
Computer Engineering Major
University of Maryland



--
   Claudio Martella
   claudio.martella@gmail.com   



--
Kyle Orlando
Computer Engineering Major
University of Maryland
--_000_CE300E0EBE3Cmajakabiljofbcom_--