Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3BB87C7DC for ; Sat, 28 Jul 2012 07:43:25 +0000 (UTC) Received: (qmail 37532 invoked by uid 500); 28 Jul 2012 07:43:24 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 37359 invoked by uid 500); 28 Jul 2012 07:43:23 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 37325 invoked by uid 99); 28 Jul 2012 07:43:22 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jul 2012 07:43:22 +0000 Received: from localhost (HELO achingmbp15.local) (127.0.0.1) (smtp-auth username aching, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jul 2012 07:43:21 +0000 Message-ID: <50139819.7020003@apache.org> Date: Sat, 28 Jul 2012 00:43:21 -0700 From: Avery Ching User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: user@giraph.apache.org Subject: Re: Heterogeneous Vertex Types References: <0E9F8A57-0AA4-4545-BD81-D9E234E69FCB@benchmarksolutions.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------010508070002090604060403" This is a multi-part message in MIME format. --------------010508070002090604060403 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Nick, Giraph needs to have be able to instantiate the vertex id, value, edges, and messages and there can only be a single type for each of them. That being said, since all the types are user chosen, you can basically implement types that support multiple types inside of it. As a simple, stupid example class MessageA { ... } class MessageB { ... } class MultiMessage implements Writable { private MessageA msgA; private MessageB msgB; private boolean useMsgA; pubic MessageA getMsgA(){ return msgA; } pubic MessageB getMsgB(){ return msgB; } ... } I didn't try and compile it, but hopefully you get the idea. =) On 7/27/12 9:52 AM, Jonathan Bishop wrote: > Nick, > > I am guessing it is because Giraph needs to move vertices around and > needs to construct them before it reads in their serialization. I am > new to Giraph myself so this may be incorrect. Maybe Avery could comment. > > Jon > > On Fri, Jul 27, 2012 at 6:32 AM, Nick West > > wrote: > > Thanks for the quick reply. > > That (using one uber-class) was my initial thought when I ran into > this problem, however I was wondering if there was a solution that > would use the type hierarchy (which admittedly is a bit more > flexible in scala than java, so this may not be possible). Is > there something in the Giraph architecture that precludes this? > > Thanks, > Nick > > On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote: > >> Nick, >> >> You may want to reconsider your approach as I don't think Giraph >> will be happy with this. >> >> How about using a new vertex value class with BasicVertex which >> can do both behaviours for you? Same for edge and message >> classes. This should be enough do what you want. >> >> Jon >> >> On Wed, Jul 25, 2012 at 3:29 PM, Nick West >> > > wrote: >> >> Hi, >> >> I'm working on implementing a belief propagation algorithm >> over Giraph. (Do you know if anyone has done this before?) >> This requires having (at least) two different types of >> vertices implemented (values and factors) and different types >> of messages sent between different vertices. >> >> I've been able to set up and run my own vertices (with my own >> custom readers and writers), however, whenever I try to >> extend this to a more complex case I run into problems. For >> example, suppose aI have the following two vertex types: >> >> class FooVertex() extends BasicVertex[IntWritable, >> IntWritable, Text, IntWritable] >> class BarVertex() extends BasicVertex[IntWritable, >> IntWritable, Text, IntWritable] >> >> (both of which run fine in the basic set up), and I then >> configure the GiraphJob with the following, >> >> job.setVertexClass(classOf[BasicVertex[IntWritable, >> IntWritable, Text, IntWritable]]) >> >> (as opposed to with FooVertex or BarVertex). >> >> I get a null pointer error in GraphMapper (at line 204 in the >> version of the code that I am using - just after the log >> statement about the input format vertex index type not being >> known, and at the line where it checks if the >> vertextIndexTypes are the same). >> >> Do you know what is causing this? (The stack trace is below.) >> Have you tried to instantiate different classes for >> different vertices in the past? >> >> One last piece of info that might explain things: I'm >> developing in scala, which I have seen behave a little bit >> poorly with java reflection - could that be a cause? >> >> Thanks, >> *Nick West >> ** >> * >> Stack Trace: >> >> ./bin/hadoop jar >> ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar >> bmrk.inference.giraph.IdiotRunner ising ising_out 4 >> Input arguments: >> [0] = ising >> [1] = ising_out >> [2] = 4 >> setting vertex class to: org.apache.giraph.graph.BasicVertex >> setting vertex input class to: >> bmrk.inference.giraph.IdiotVertexInputFormat >> setting vertex output class to: >> bmrk.inference.giraph.IdiotVertexOutputFormat >> 12/07/25 17:48:54 INFO mapred.JobClient: Running job: >> job_201207250932_0012 >> 12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0% >> 12/07/25 17:52:43 INFO mapred.JobClient: Task Id : >> attempt_201207250932_0012_m_000000_0, Status : FAILED >> java.lang.NullPointerException >> at >> org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204) >> at >> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392) >> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655) >> at >> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> >> >> Benchmark Solutions >> 101 Park Avenue - 7th Floor >> New York, NY 10178 >> Tel +1.212.220.4739 | Mobile >> +1.646.267.4324 >> *www.benchmarksolutions.com * >> >> * >> >> >> * >> >> > > * > Nick West > * > * > *Benchmark Solutions > 101 Park Avenue - 7th Floor > New York, NY 10178 > Tel +1.212.220.4739 | Mobile > +1.646.267.4324 > *www.benchmarksolutions.com * > * > > > * > > --------------010508070002090604060403 Content-Type: multipart/related; boundary="------------030605050604000709010505" --------------030605050604000709010505 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Hi Nick,

Giraph needs to have be able to instantiate the vertex id, value, edges, and messages and there can only be a single type for each of them.

That being said, since all the types are user chosen, you can basically implement types that support multiple types inside of it.  As a simple, stupid example

class MessageA {
...
}

class MessageB {
...
}

class MultiMessage implements Writable {
    private MessageA msgA;
    private MessageB msgB;

    private boolean useMsgA;

    pubic MessageA getMsgA(){
        return msgA;
    }

    pubic MessageB getMsgB(){
        return msgB;
    }

    ...
}

I didn't try and compile it, but hopefully you get the idea.  =)

On 7/27/12 9:52 AM, Jonathan Bishop wrote:
Nick,

I am guessing it is because Giraph needs to move vertices around and needs to construct them before it reads in their serialization. I am new to Giraph myself so this may be incorrect. Maybe Avery could comment.

Jon

On Fri, Jul 27, 2012 at 6:32 AM, Nick West <nick.west@benchmarksolutions.com> wrote:
Thanks for the quick reply.

That (using one uber-class) was my initial thought when I ran into this problem, however I was wondering if there was a solution that would use the type hierarchy (which admittedly is a bit more flexible in scala than java, so this may not be possible).  Is there something in the Giraph architecture that precludes this?

Thanks,
Nick

On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:

Nick,

You may want to reconsider your approach as I don't think Giraph will be happy with this.

How about using a new vertex value class with BasicVertex which can do both behaviours for you? Same for edge and message classes. This should be enough do what you want.

Jon

On Wed, Jul 25, 2012 at 3:29 PM, Nick West <nick.west@benchmarksolutions.com> wrote:
Hi,

I'm working on implementing a belief propagation algorithm over Giraph.  (Do you know if anyone has done this before?)  This requires having (at least) two different types of vertices implemented (values and factors) and different types of messages sent between different vertices.

I've been able to set up and run my own vertices (with my own custom readers and writers), however, whenever I try to extend this to a more complex  case I run into problems.  For example, suppose aI have the following two vertex types:

class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]

(both of which run fine in the basic set up), and I then configure the GiraphJob with the following,

job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text, IntWritable]])

(as opposed to with FooVertex or BarVertex). 

I get a null pointer error in GraphMapper (at line 204 in the version of the code that I am using - just after the log statement about the input format vertex index type not being known, and at the line where it checks if the vertextIndexTypes are the same).   

Do you know what is causing this? (The stack trace is below.)  Have you tried to instantiate different classes for different vertices in the past?  

One last piece of info that might explain things: I'm developing in scala, which I have seen behave a little bit poorly with java reflection - could that be a cause?

Thanks,
Nick West
Stack Trace:

./bin/hadoop jar ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar bmrk.inference.giraph.IdiotRunner ising ising_out 4
Input arguments:
[0] = ising
[1] = ising_out
[2] = 4
setting vertex class to: org.apache.giraph.graph.BasicVertex
setting vertex input class to: bmrk.inference.giraph.IdiotVertexInputFormat
setting vertex output class to: bmrk.inference.giraph.IdiotVertexOutputFormat
12/07/25 17:48:54 INFO mapred.JobClient: Running job: job_201207250932_0012
12/07/25 17:48:55 INFO mapred.JobClient:  map 0% reduce 0%
12/07/25 17:52:43 INFO mapred.JobClient: Task Id : attempt_201207250932_0012_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)


Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178 
Tel +1.212.220.4739 | Mobile +1.646.267.4324 
www.benchmarksolutions.com 
<image001.png> 
 




Nick West

Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178 
Tel +1.212.220.4739 | Mobile +1.646.267.4324 
www.benchmarksolutions.com 
 
 



--------------030605050604000709010505 Content-Type: image/png Content-Transfer-Encoding: base64 Content-ID: iVBORw0KGgoAAAANSUhEUgAAAPQAAAAmCAYAAADz2bbaAAAAAXNSR0IArs4c6QAAAARnQU1B AACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAZdEVYdFNvZnR3YXJlAE1pY3Jvc29m dCBPZmZpY2V/7TVxAAALC0lEQVR4Xu2dsYskRRTG7z9RMFYwVjBWTEUxEkERMVMvPsVEk0MO E+FEowNPxECDOwRBhFMDMfAOExE5Iw+DAwWzkd/AN3z7zavqnt3Z3dmdahhuu7uquurV+973 3qvqvguLcQwJDAmcGwlcODcjGQMZEhgSWBwa0L//cW/xy69/LvTvkOWQwJDA6UtgNqD/+fe/ xadf/LC49O7nizcvXVs88+KV5d8fX/t2de2b7+4s3rvy1eKve/dPf2SjB0MCeyiBWYAGqAAX Rv7y5s9LMANu/v7g6tdLEOsAzJTnOuXHMSQwJHByEugCWiB+9eInSxAD0Bde+3DJ0DC2GJlr nPtBXYE6753c8MaThgT2SwJNQAuQiAPgAmoYGWaGkQExIOc6ZSnDfRga4KsM4JcB2C/RjtEO CZy8BNYADSCJjQGrfu5S08Uff/ptCVj+1cE55blGfQBPwgwDIBf95Ic3njgksF8SWAM0bAsA xbhVLMw9JcVgZRhZ4Me9Bsi6jzj5G4DT1nk97t+/v/j+1q3V7yyP887t26tx3L17d6eH4jJn Dvb9OABowKysNYKBXQEi13TA4JwrrnYm5xpgVl0ZBk+iYQzO44FiPfTAg6vfWR7j888+txrH +5cvb20obiiQ1zYOl/m22mz167jksg05qI0DgAa8/GBdX5rCneZc7A1wOQC3st4ObMoq7oaV FX9zDSPRO1xoPln6+5WXXl7cvHFjmzLYSlsD0LUYYc133np78ejDjxwweJrPi6+/cST5Hxeg K8/kzAFaSS0ltDjnEBvDvkqGiWkBtcDshkDsrNkC2HK9e1nvKUBrAlGSXToGoNdnAzA//eRT JZA1j8z3UY5tApr+fnT16qrP2a8zA2gA5vEtMbE2h5DkAohKjCnJJZYWQ5MI00Fd2pP7LTbX phN34XtCw6ojRH5PPPb4mmLsUsw0AL0OS9x1BxxzyDV+GGTAflTDvE1AT83hWcgtLF1uuc7O prjNgFXZazEy4HeWlmsuNteOMsqkey133sHfA3Ra71SQVswkwW+a0FGCZYoxaFdlZVRayqBk GX2ac6jtylipralxef/mPJMy/txtMZG3g3He1ACrTz3ZnSSg58jSk6NzynsZ6e1UPeleJZcl oLUDTA1VCS9nXK1Fa82Zcy1RCezKcmu3GID3dltutytBAjpB44BGkMRjGXeLFVxIXoZ4vKpX MQfuWOUpENdn3xB2hg/UVfw/pzz1GVerLZ9Q2qUfrbyDl/UytO/16FcL0DzD6065yx43T5XV /GgeM+bmnHlKY1YB2q9lUq/SoakwT33vGbrPrl8vdYM6STzeDvfoY46XsboBrHRA4+TZOpaA hoX5AUoYF6BqPdmTXdpI4uvPasiXrmBgzuWua11bS2G003K7e4BO4GnAc2I1T774hLeSNZRx ZahA73FgKkqvXTHPnH7glrbawkDoSO8lgU0bAoPfw3AlKCrFRaG8H/RrinETKBjE3pHPqIwT fWgZJwHnpAHdMqStfrhcKoLIhOGUXFxPl4BWMkvxsNxnJbK0a0zs23r5QltF3XXXllHYGWPB 4TF5z+X2GDqV2q2Sg41yKI4s35QF577Hdv4cAYZnJfgAgmJB7iegKc/EUSaNAedVeRSD8lUi SfdSeaTEsvL0CyblOnLw8cjrqICiaxVDp8F049ADaGVkJOsqbEjlRk6SccpfxmRqfucwNPNX zZPifelaZejSIGrOqZvjkXeWhg556lmp55XnSVnmifY03wcYWuzKRYCnhJdcZHePe+vI3HP2 Vn3aVTJN+8E9YeZKMeX+MGAHMwPuxVEucLF0KocrFyBIl9QBlgyhvidA08VMzyPLA1QdWGTv gzNxjlcKS3sVY7oxUp8S0M6ctJGK60akNf4WsHvz6W5lGs1k85SX7m8D0K057JENcs+5yDnn voMaPeJImbhLnmFNGli10ZL3kqG1VVNsDYNm9lqZa+61GFoAhsmVTFN2XMtbPItfaz06Eymc V26J2MYnGmVTwkD/OkAlDFeCXAetYqx0USthZr1cK3e2Ulzl7Wb5ntvo8nAGUkJG1j5ZpwJ0 pSA5B94XN6Zd/9lutuJL2pXr7kbDDZg/ww1rNZbDutyHBXSCr0pSVQTh8q3kn0YqPTx5Oc2k mMCLS6y4V0ksgKcdY4qnq/gX5q12kGmnGYwMg8P2eu2yUoheDJ3CQVHyWs+d5B6Hl5lignQb W5taEtDJllOAzsRJD+yV61clVlIWFQiqnWAtVk0GmgtolUN2Vdv0oTfvqu/eVmWcTxrQqRtz DH0v6aj6CWg8yFYeBbm5rq2y3Nolphcs1Lh2h3nWGlDCxgBdL3P4K5Tay50JNWW8ud9i+amJ TSt9VEAnkCrXrgeuudb9KIDuZUlpN2WABVeM79b9MIBORaoMwKbATveafk3NO89wFt8Fhk5A VyHPVFa9kmcVRgBq5rLyVt3QLgENQHGD5VZ7zMw13Gb/WIFezlCSS6623GgBWplzGNnXnjmf E0NXjJCATpd7jnL1Yu4pl7u1VTHrZT+OE9AuE4/F6cNUDD3F0MmeyC4NzByZZxlXTOY5k30V OLJOeltzGDqNiY9lag7TM0qXu/LeMmlW5ShSNj39pCz9zMSp2ljt5YaJcbcBZya+9KaUQK34 WNlwAVws7W9s8SDq+U60uTvFHNDalueDRQkyMdFaGvGJ2xTQ6SZWscuUMhwnoOe657BtgmAO oNPlm5vlbm2USFAxj3kt+9VKmlVz2Vr/ziRVGqecw8zEJ6Bpr7eUV91H/lXI5KCeAjRl0ytb AzSsLBcZNhWDAlbfLOKbQ9xtdpaW+643s7R+rTi792miKinWih9kETNpQBsogNL6suytGMWF WTF0XtMmByWdlIzziTgthmasTDbjr+LVwwA6mV7JrClmlhGjT/RF2z0ztleiLVmHOtVyEu31 lq1SH7TkV7mrPYamP0owtoCYbrcSVtULKXrWpoDmGeoLbWS+RIaaPh542wqG1u4vrRUrpgbY +eEDQAzzUk9vV2nziH/VhL+VZKONbbyckUstUy8BKCHWctOknBWguZeuUyrlaTJ0un7et8o1 62XQewySbvHUm1Kp7FXC0tuAEXsbLag/Z2NJLvv5c5McMnyoni/AtIDY23SkZ/vqwGEA3Uv2 ru0UkzKLQZX11poxwBUItdlEe7jF6lq7VlsYBq6JjeXKy01vWXeAoyRJ9S9K0trL7G/KpFK7 C+ftpvusLXYqk9srK8OBEmS9HJ8YU0zVe45AVfVBxkX3NJlVXKV1Xh9vtl0tQ/kc5Jo/oO7J z8fdymozN9oum3KCeZmrBBbnuR2yJyeeXW1UUQzbki26lZ6NAN2SizyY1oagNBq9dqoxtZb9 KhmufbFEgAWoHP6VT859txggh52pk+vKAJqy/tnf/JTRlMs27g8JDAlsJoHyI4ECMUBVltu/ PqLstt6sgn3z9Un/Xrf2bo+vf242OaP0kMCmEigBDfAAre/2Atj6dK8+EujuNgxNHTG8XHDa mPpKyaadHuWHBIYEagk0P+Pr7rTcb29C3x9TNlwJM871N0CmHMZAybbxv2oMVRwSOD4JTP7P GSS1AKb+Hyt1xfdka91ZG1CU9VZZvY6p96aPbzij5SGB/ZbAJKAdlFqiwp3W10fcLde+78HC +61UY/SnJ4HZgM4uwthi7dY2ztMb1njykMB+SuDQgN5PcY1RDwnstgQGoHd7fkbvhgQ2ksAA 9EbiGoWHBHZbAgPQuz0/o3dDAhtJAED/PX5DBkMHzocO/A8padvYOoPF2AAAAABJRU5ErkJg gg== --------------030605050604000709010505-- --------------010508070002090604060403--