hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser" <josh.el...@gmail.com>
Subject Re: Review Request 23914: Add AccumuloStorageHandler
Date Tue, 29 Jul 2014 22:34:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23914/
-----------------------------------------------------------

(Updated July 29, 2014, 10:34 p.m.)


Review request for hive.


Changes
-------

Fixes minor issue with serialization of strings as compared to what HBaseStorageHandler does.
Allows binary encoding to be specified on strings without error (falls back to UTF8 serialization)


Bugs: HIVE-7068
    https://issues.apache.org/jira/browse/HIVE-7068


Repository: hive-git


Description
-------

Adds an AccumuloStorageHandler which allows Accumulo tables to back Hive tables. Predicate
pushdown, table range elimination, UTF8 and binary serialization, and more.


Diffs (updated)
-----

  accumulo-handler/pom.xml PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/AccumuloConnectionParameters.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/AccumuloHiveConstants.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/AccumuloHiveRow.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/AccumuloStorageHandler.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/LazyAccumuloMap.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/LazyAccumuloRow.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/Utils.java PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnEncoding.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMapper.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMapping.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMappingFactory.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/HiveAccumuloColumnMapping.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/HiveAccumuloMapColumnMapping.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/HiveAccumuloRowIdColumnMapping.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/HiveColumn.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/InvalidColumnMappingException.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloRecordReader.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloSplit.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloTableInputFormat.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/HiveAccumuloTableOutputFormat.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/package-info.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloPredicateHandler.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/AccumuloRangeGenerator.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/NoSuchCompareOpException.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/NoSuchPrimitiveComparisonException.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PushdownTuple.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/CompareOp.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/DoubleCompare.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/Equal.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/GreaterThan.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/GreaterThanOrEqual.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/IntCompare.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/LessThan.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/LessThanOrEqual.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/Like.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/LongCompare.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/NotEqual.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/PrimitiveComparison.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/StringCompare.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/compare/package-info.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/package-info.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloCompositeRowId.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowIdFactory.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowSerializer.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDe.java PRE-CREATION

  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/CompositeAccumuloRowIdFactory.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/TooManyAccumuloColumnsException.java
PRE-CREATION 
  accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/TooManyHiveColumnsException.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestAccumuloConnectionParameters.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestAccumuloHiveRow.java PRE-CREATION

  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestAccumuloStorageHandler.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloMap.java PRE-CREATION

  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloRow.java PRE-CREATION

  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/columns/TestColumnEncoding.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/columns/TestColumnMapper.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/columns/TestColumnMappingFactory.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/columns/TestHiveAccumuloColumnMapping.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/columns/TestHiveRowIdColumnMapping.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableInputFormat.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableOutputFormat.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTypes.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/TestAccumuloPredicateHandler.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/TestAccumuloRangeGenerator.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/TestPrimitiveComparisonFilter.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/compare/TestDoubleCompare.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/compare/TestIntCompare.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/compare/TestLongComparison.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/predicate/compare/TestStringCompare.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/DelimitedAccumuloRowIdFactory.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/FirstCharAccumuloCompositeRowId.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloRowSerializer.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDe.java PRE-CREATION

  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDeParameters.java
PRE-CREATION 
  accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestDefaultAccumuloRowIdFactory.java
PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_custom_key.q PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_custom_key2.q PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_joins.q PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_predicate_pushdown.q PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_queries.q PRE-CREATION 
  accumulo-handler/src/test/queries/positive/accumulo_single_sourced_multi_insert.q PRE-CREATION

  accumulo-handler/src/test/results/positive/accumulo_custom_key.q.out PRE-CREATION 
  accumulo-handler/src/test/results/positive/accumulo_custom_key2.q.out PRE-CREATION 
  accumulo-handler/src/test/results/positive/accumulo_joins.q.out PRE-CREATION 
  accumulo-handler/src/test/results/positive/accumulo_predicate_pushdown.q.out PRE-CREATION

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out PRE-CREATION 
  accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out PRE-CREATION

  accumulo-handler/src/test/templates/TestAccumuloCliDriver.vm PRE-CREATION 
  itests/qtest/pom.xml 249956f 
  itests/util/pom.xml aca01cb 
  itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloQTestUtil.java PRE-CREATION

  itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloTestSetup.java PRE-CREATION

  packaging/pom.xml cc12d1b 
  pom.xml b5a5697 

Diff: https://reviews.apache.org/r/23914/diff/


Testing
-------

Lots of unit tests. QTests added which use MiniAccumuloCluster. Real queries run against an
actual Accumulo instance. All of the former performed against both Accumulo 1.5.1 and Accumulo
1.6.0 (last two "active" versions).


Thanks,

Josh Elser


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message