giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Armando" <a.mirag...@student.vu.nl>
Subject Review Request 14068: GIRAPH-723: Rexster Output Format
Date Tue, 10 Sep 2013 20:14:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14068/
-----------------------------------------------------------

Review request for giraph.


Repository: giraph-git


Description
-------

This patch provides a way to use Rexster as output format endpoint. By means of ZooKeeper
coordination, the workers will first write all the vertices and then all the edges. Such an
order is required by the bluprint API.

Limitations:
- The current approach does not yet work with neo4j. In fact, neo4j does not properly handle
saving vertices not in order of ID. I am looking into this currently.
- This approach does not scale. In fact, the batch-kibble provided by Rexster and used by
the output format loads the whole graph in memory before committing the data collected into
the underlining database. This will be solved with a Giraph provided kibble to stream the
data from the output format.


Diffs
-----

  giraph-rexster/src/main/java/org/apache/giraph/rexster/conf/GiraphRexsterConstants.java
c4b4655 
  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeInputFormat.java 4dee078

  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterEdgeOutputFormat.java PRE-CREATION

  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexInputFormat.java
da45b5b 
  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/RexsterVertexOutputFormat.java
PRE-CREATION 
  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatEdgeOutputFormat.java
PRE-CREATION 
  giraph-rexster/src/main/java/org/apache/giraph/rexster/io/formats/RexsterLongDoubleFloatVertexOutputFormat.java
PRE-CREATION 
  giraph-rexster/src/main/java/org/apache/giraph/rexster/utils/RexsterUtils.java e669ca9 
  giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestAbstractRexsterInputFormat.java
09b27d3 
  giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatInputFormat.java
PRE-CREATION 
  giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatOutputFormat.java
PRE-CREATION 
  giraph-rexster/src/test/java/org/apache/giraph/rexster/io/formats/TestRexsterLongDoubleFloatVertexInputFormat.java
a68db67 

Diff: https://reviews.apache.org/r/14068/diff/


Testing
-------

tested on a small cluster using tinkergraph as database exported with Rexster.


Thanks,

Armando


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message