giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Review Request: GIRAPH-588: More flexible Hive input
Date Mon, 25 Mar 2013 21:53:06 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10126/
-----------------------------------------------------------

Review request for giraph and Nitay Joffe.


Description
-------

Currently with Hive input formats it's only possible to convert single row to single vertex/edge.
We should support some rows to be skipped. General multiple rows to multiple vertices/edges
conversion can be useful (one example - we have edge input, sorted by source id, and we want
to keep only top k weighted edges per vertex).

Added some wrappers for readers to convert them to Iterator-like classes.


This addresses bug GIRAPH-588.
    https://issues.apache.org/jira/browse/GIRAPH-588


Diffs
-----

  giraph-core/src/main/java/org/apache/giraph/io/iterables/EdgeReaderWrapper.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/EdgeWithSource.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/GiraphReader.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/IteratorToReaderWrapper.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/VertexReaderWrapper.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/package-info.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java efc08d3 
  giraph-hive/src/main/java/org/apache/giraph/hive/common/DefaultConfigurableAndTableSchemaAware.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/RecordReaderWrapper.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/AbstractHiveToEdge.java f29fea7

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeInputFormat.java 18b40c2

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java 6fb183a

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveToEdge.java 2205b82 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/SimpleHiveToEdge.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/AbstractHiveToVertex.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/AbstractHiveToVertexEdges.java
d0668f6 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/AbstractHiveToVertexValue.java
9ab316f 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertex.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertexEdges.java 8076a8a

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertexValue.java 593eb9a

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexInputFormat.java
25c7a26 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexReader.java 541176f

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/SimpleHiveToVertex.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/SimpleNoEdgesHiveToVertex.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/output/AbstractVertexToHive.java 8e3f1ca


Diff: https://reviews.apache.org/r/10126/diff/


Testing
-------


Thanks,

Maja Kabiljo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message