giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <olivier.var...@orange.com>
Subject BUG ? SequenceFileVertexInputFormat with custom Writable : can not iterate over vertices
Date Thu, 04 Sep 2014 09:46:34 GMT
Hi,

first of all many thanks to the community for the great work you are doing.


with giraph 1.1-SNAPSHOT
built for hadoop 2.4.0


I am stalled on a bug using custom SequenceFile Vertex Input format.


I have opened a project on github to allow you to run/play/view the bug

https://github.com/ovarene/giraph_SequenceFileVertexInput


you will find everything needed :D



I try to load/create my vertices from a SequenceFile<LongWritable,MyWritable>
"LongWritable" being the vertexId
"MyWritable" being a custom value writable used to generate VertexValue and its edges


Vertices are created correctly and loaded (as seen on the log)

2014-09-04 10:44:44,683 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108))
- Create Vertex from "1.0 0.0 0:" to : Vertex(id=1,value=1.0,#edges=0) 2014-09-04 10:44:44,694
DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) -
Create Vertex from "2.0 0.0 2:1|6|" to : Vertex(id=2,value=2.0,#edges=2) 2014-09-04 10:44:44,695
DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) -
Create Vertex from "5.0 0.0 3:4|1|6|" to : Vertex(id=5,value=5.0,#edges=3)

2014-09-04 10:44:44,697 INFO [load-0] worker.InputSplitsCallable (InputSplitsCallable.java:call(235))
- call: Loaded 1 input splits in 0.12198963 secs, (v=3, e=5) 24.592255 vertices/sec, 40.98709
edges/sec


But alas, when trying to iterate over them during a compute session, I get the following error

ERROR : But when reading in PrintVertex (BasicComputation): 2014-09-04 10:44:44,701 ERROR
[load-0] utils.LogStacktraceCallable (LogStacktraceCallable.java:call(57)) - Execution of
callable failed java.lang.IllegalStateException: next: IOException at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:101)
at org.apache.giraph.partition.BasicPartition.addPartitionVertices(BasicPartition.java:99)
at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482)
at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:428)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:241) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: ensureRemaining:
Only 5 bytes remaining, trying to read 8 at org.apache.giraph.utils.UnsafeByteArrayInputStream.ensureRemaining(UnsafeByteArrayInputStream.java:114)
at org.apache.giraph.utils.UnsafeByteArrayInputStream.readLong(UnsafeByteArrayInputStream.java:197)
at org.apache.hadoop.io.LongWritable.readFields(LongWritable.java:47) at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:522)
at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:98) ... 11 more


Any idea ?

Thanks a lot

Olivier










_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees
et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par
erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant
susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may
be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message
and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed
or falsified.
Thank you.


Mime
View raw message