giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nitay Joffe" <ni...@apache.org>
Subject Re: Review Request: GIRAPH-453: Pure Hive I/O (nitay)
Date Sat, 15 Dec 2012 01:21:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8611/
-----------------------------------------------------------

(Updated Dec. 15, 2012, 1:21 a.m.)


Review request for giraph.


Description (updated)
-------

GIRAPH-453: Pure Hive I/O (nitay)

For now this is only the Input side of things. One particular thing I added was the concept
of "profiles", allowing for easily readingĀ from multiple tables. This should remove a lot
of the cruft around the GiraphHCat* classes.

In terms of performance this is on par with our current HCatalog stuff. I ran a few jobs and
noticed at most a few seconds of difference between the input supersteps.


This addresses bug GIRAPH-453.
    https://issues.apache.org/jira/browse/GIRAPH-453


Diffs
-----

  giraph-formats-contrib/pom.xml 9326c28420be2749fa724ebd86d9555ae209a2ee 
  giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java
PRE-CREATION 
  giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HiveGiraphRunner.java
7a7c2f87bc50eaebe879e2eff2da661b504096b0 
  giraph-hive/pom.xml PRE-CREATION 
  giraph-hive/src/main/assembly/compile.xml PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveConstants.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveEdgeInputFormat.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveEdgeReader.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveUtils.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexCreator.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexInputFormat.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/HiveVertexReader.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/VertexCreator.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveApiInputFormat.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveApiOutputFormat.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveInputDescription.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveOutputDescription.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/HiveRecord.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/Classes.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/HadoopUtils.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/ProgressReporter.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/SerDes.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/common/Writables.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiInputSplit.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiRecord.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/ApiRecordReader.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputConf.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputPartition.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputPartitions.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/input/InputSplitData.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/ListParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/MapParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/ObjectParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/Parsers.java PRE-CREATION 
  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/PrimitiveParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/StructParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/inspect/UnionParser.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/output/OutputConf.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedField.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedList.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedMap.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedPrimitive.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedSchema.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedStruct.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/hadoop/hive/api/schema/MaterializedUnion.java PRE-CREATION

  pom.xml 47736351c45f4cdb55df762506069eaed7126a7e 

Diff: https://reviews.apache.org/r/8611/diff/


Testing
-------


Thanks,

Nitay Joffe


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message