hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anthony Hsu" <a...@linkedin.com>
Subject Re: Review Request 20096: HIVE-6835: Reading of partitioned Avro data fails if partition schema does not match table schema
Date Thu, 17 Apr 2014 01:14:54 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20096/
-----------------------------------------------------------

(Updated April 17, 2014, 1:14 a.m.)


Review request for hive.


Changes
-------

Addressed Ashutosh's comments in HIVE-6835. Added the constant to serde.thrift and used the
Thrift compiler to generate all the language-specific bindings.


Repository: hive-git


Description
-------

The problem occurs when you store the "avro.schema.(literal|url)" in the SERDEPROPERTIES instead
of the TBLPROPERTIES, add a partition, change the table's schema, and then try reading from
the old partition.

I fixed this problem by passing the table properties to the partition with a "table." prefix,
and changing the Avro SerDe to always use the table properties when available.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 43cef5c 
  ql/src/test/queries/clientpositive/avro_partitioned.q 6fe5117 
  ql/src/test/results/clientpositive/avro_partitioned.q.out 644716d 
  serde/if/serde.thrift 31c87ee 
  serde/src/gen/thrift/gen-cpp/serde_constants.h d56c917 
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp 54503e3 
  serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java 515cf25

  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php 837dd11 
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 8eac87d 
  serde/src/gen/thrift/gen-rb/serde_constants.rb ed86522 
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 9d58d13 
  serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 67d5570 

Diff: https://reviews.apache.org/r/20096/diff/


Testing
-------

Added test cases


Thanks,

Anthony Hsu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message