hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoltan Haindrich <k...@rxd.hu>
Subject Re: Review Request 50586: HIVE-14329 fix flapping qtests
Date Sat, 30 Jul 2016 20:22:06 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50586/
-----------------------------------------------------------

(Updated July 30, 2016, 8:22 p.m.)


Review request for hive.


Changes
-------

add new version of patch


Summary (updated)
-----------------

HIVE-14329 fix flapping qtests


Bugs: HIVE-14329
    https://issues.apache.org/jira/browse/HIVE-14329


Repository: hive-git


Description (updated)
-------

fix hashmap order changes inside qtest outputs; most common is I think is skewed key order
differences.
main fix is some added sorting logic to MetaDataFormatUtils
i've seen some STATS_ACCURATE / COLUMN_STATS related order changes too; in that class i've
noticed some minor problems:

* fix for StatsSetupConst#setBasicStatsState: failed to upgrade&set basic stats status
* removed: inconsitency: StatsSetupConst#setColumnStatsState would put lowercase colnames
if record is new; original colnames during update/addition
  i've choosed to be keep the column names **case sensitive** - this is good?
* removed: upgrade-while-read from: StatsSetupConst#areBasicStatsUptoDate

even thru this may help avoiding STATS_ACCURATE related problems; it's not bulletproof - jsonobject
tries hard to keep the insertion order inside its parts.

i'm not sure but: ...would it make sense to use jackson/ObjectMapper to parse the params into
a pojo; which then can be used more easily to read/write colstats and other metadata like
it more easily - i haven't looked into it; how complicated would this be...


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 01e6010 
  common/src/test/org/apache/hadoop/hive/common/TestStatsSetupConst.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java 03803bb

  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_1.q.out 00f3776 
  ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out df4a75b 
  ql/src/test/results/clientpositive/create_skewed_table1.q.out fe5ea0f 
  ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out 8f40040 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 0acfe90 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial_ndv.q.out 70e4db3 
  ql/src/test/results/clientpositive/list_bucket_dml_10.q.out d4681b7 
  ql/src/test/results/clientpositive/list_bucket_dml_11.q.out ecf54a8 
  ql/src/test/results/clientpositive/list_bucket_dml_12.q.out 0e11f3f 
  ql/src/test/results/clientpositive/list_bucket_dml_13.q.out 93ebef0 
  ql/src/test/results/clientpositive/list_bucket_dml_2.q.out a29c224 
  ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 5f0406a 
  ql/src/test/results/clientpositive/list_bucket_dml_5.q.out 09cb847 
  ql/src/test/results/clientpositive/list_bucket_dml_6.q.out e53fee7 
  ql/src/test/results/clientpositive/list_bucket_dml_8.q.out ee36d3f 
  ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 81f3af3 
  ql/src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out 492866a 
  ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out cfb613a 
  ql/src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out 6a5f8f5 
  ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out 0c6fb63 
  ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.out 9eca85a 
  ql/src/test/results/clientpositive/stats_list_bucket.q.out c66da97 

Diff: https://reviews.apache.org/r/50586/diff/


Testing (updated)
-------

r2: added qtest changes to patch


Thanks,

Zoltan Haindrich


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message