hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bkm.had...@gmail.com
Subject Review Request: HIVE-1634 - Allow access to Primitive types stored in binary format in HBase
Date Mon, 13 Sep 2010 21:10:58 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/826/
-----------------------------------------------------------

Review request for Hive Developers and John Sichi.


Summary
-------

This addresses HIVE-1245 in part, for atomic or primitive types.

The serde property "hbase.columns.storage.types" = "-,b,b,b,b,b,b,b,b" is a specification
of the storage option for the corresponding column in the serde property "hbase.columns.mapping".
Allowed values are '' for table default, 's' for standard string storage, and 'b' for binary
storage as would be obtained from o.a.h.hbase.utils.Bytes. Map types for HBase column families
use a colon separated pair such as 's:b' for the key and value part specifiers respectively.
See the test cases and queries for HBase handler for additional examples.

There is also a table property "hbase.table.default.storage.type" = "string" to specify a
table level default storage type. The other valid specification is "binary". The table level
default is overridden by a column level specification.

This control is available for the boolean, tinyint, smallint, int, bigint, float, and double
primitive types. The attached patch also relaxes the mapping of map types to HBase column
families to allow any primitive type to be the map key.


This addresses bug HIVE-1634.
    http://issues.apache.org/jira/browse/HIVE-1634


Diffs
-----

  trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 990439 
  trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java 990439 
  trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 990439 
  trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 990439 
  trunk/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 990439

  trunk/hbase-handler/src/test/queries/hbase_binary_map_queries.q PRE-CREATION 
  trunk/hbase-handler/src/test/queries/hbase_binary_storage_queries.q PRE-CREATION 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBooleanBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyByteBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyDoubleBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 990439 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFloatBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyIntegerBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyLongBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyShortBinary.java PRE-CREATION

  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 990439 

Diff: http://review.cloudera.org/r/826/diff


Testing
-------

The HBase handler tests TestHBaseSerDe, TestLazyHBaseObject, TestHBaseCliDriver, and TestHBaseMinimrCliDriver
pass.

New tests have been added to TestHBaseSerDe and TestLazyHBaseObject to test this feature.

New queries which exercise this feature have been added to query files hbase_binary_map_queries.q
and hbase_binary_storage_queries.q.


Thanks,

bkm


Mime
View raw message