incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jake Mannix (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
Date Fri, 16 Sep 2011 19:38:09 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106748#comment-13106748
] 

Jake Mannix commented on GIRAPH-34:
-----------------------------------

I'll definitely open another JIRA for the Vertex subclasses, and dig into that a bit.

But on this current topic, I see how users could possibly do something like sendMsg(destVertex,
getVertexValue()), yes.  But isn't this analogous to in regular Hadoop-land, that you simply
cannot expect to hang onto your Writable instances and use them later.  If you're in Mapper.map(SomethingWritableComparable
key, SomethingWritable value, Context c), you should *never* just buffer up the key and value
instances, as this is practically guaranteed to break - Hadoop will be re-using the key and
value as container objects to read new bytes off of disk for the next invocation to map(),
so that java objects are rarely created, instead you're just constantly doing simple bit/byte
operations on the disk stream, and setting values inside of Writable containers.  

It seems like one of the basic contracts of Writables (at least in Hadoop-land) is that they
are always to be considered containers: call get() or getSomeKindOfThing() on them as soon
as you have a handle on one, and use whatever *that* is, assuming that the framework can and
will reuse your original Writable.

> Failure of Vertex reflection for putVertexList from GIRAPH-27 
> --------------------------------------------------------------
>
>                 Key: GIRAPH-34
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-34
>             Project: Giraph
>          Issue Type: Bug
>            Reporter: Christian Kunz
>            Assignee: Avery Ching
>         Attachments: GIRAPH-34.patch
>
>
> Christian actually found this bug.  I am filing the JIRA on his behalf.  Here's my error
when running TestVertexRangeBalancer.  
> java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002
failed on local exception: java.io.EOFException
> 	at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768)
> 	at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282)
> 	at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on
local exception: java.io.EOFException
> 	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1033)
> 	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
> 	at $Proxy3.putVertexList(Unknown Source)
> 	at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766)
> 	... 10 more
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:375)
> 	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
> I identified and fixed the issue by making BasicVertex implement Configurable and making
the graph state set in BasicRPCCommunications.  There is one more error though that I'll try
and solve before putting up a reviewboard.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message