hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pengcheng xiong" <pxi...@hortonworks.com>
Subject Re: Review Request 34576: Bucketized Table feature fails in some cases
Date Fri, 29 May 2015 18:15:24 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34576/
-----------------------------------------------------------

(Updated May 29, 2015, 6:15 p.m.)


Review request for hive and John Pullokkaran.


Repository: hive-git


Description
-------

Bucketized Table feature fails in some cases. if src & destination is bucketed on same
key, and if actual data in the src is not bucketed (because data got loaded using LOAD DATA
LOCAL INPATH ) then the data won't be bucketed while writing to destination.
Example
----------------------------------------------------------------------
CREATE TABLE P1(key STRING, val STRING)
CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH '/Users/jp/apache-hive1/data/files/P1.txt' INTO TABLE P1;
– perform an insert to make sure there are 2 files
INSERT OVERWRITE TABLE P1 select key, val from P1;
--------------------------------------------------
This is not a regression. This has never worked.
This got only discovered due to Hadoop2 changes.
In Hadoop1, in local mode, number of reducers will always be 1, regardless of what is requested
by app. Hadoop2 now honors the number of reducer setting in local mode (by spawning threads).
Long term solution seems to be to prevent load data for bucketed table.


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java e53933e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1a9b42b 
  ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
  ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_1.q.out f4522d2 
  ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_2.q.out 9aa9b5d 
  ql/src/test/results/clientnegative/exim_11_nonpart_noncompat_sorting.q.out 9220c8e 
  ql/src/test/results/clientpositive/auto_join32.q.out f862870 
  ql/src/test/results/clientpositive/auto_join_filters.q.out a6720d9 
  ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 5114038 
  ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out e6e7ef3 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out b2e782f 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 210f1ab 
  ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out a307b13 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out f4ceee7 
  ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 3c2951a 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e1f3888 
  ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 38ecdbe 
  ql/src/test/results/clientpositive/bucket_map_join_1.q.out 42e6a3f 
  ql/src/test/results/clientpositive/bucket_map_join_2.q.out af73309 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 870ecdd 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 33f5c46 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out 067d1ff 
  ql/src/test/results/clientpositive/bucketcontext_1.q.out 77bfcf9 
  ql/src/test/results/clientpositive/bucketcontext_2.q.out a9db13d 
  ql/src/test/results/clientpositive/bucketcontext_3.q.out 9ba3e0c 
  ql/src/test/results/clientpositive/bucketcontext_4.q.out a2b37a8 
  ql/src/test/results/clientpositive/bucketcontext_5.q.out 3ee1f0e 
  ql/src/test/results/clientpositive/bucketcontext_6.q.out d2304fa 
  ql/src/test/results/clientpositive/bucketcontext_7.q.out 1a105ed 
  ql/src/test/results/clientpositive/bucketcontext_8.q.out 138e415 
  ql/src/test/results/clientpositive/bucketizedhiveinputformat_auto.q.out 215efdd 
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out 471ff73 
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out b0e849d 
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out 4263cab 
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out bcd7394 
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out a8d9e9d 
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out c759f05 
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out f61500c 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out 0cb2825 
  ql/src/test/results/clientpositive/bucketmapjoin7.q.out 667a9db 
  ql/src/test/results/clientpositive/bucketmapjoin8.q.out 252b377 
  ql/src/test/results/clientpositive/bucketmapjoin9.q.out 5e28dc3 
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 6ae127d 
  ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 4c9f54a 
  ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 9a0bfc4 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 34cd1ff 
  ql/src/test/results/clientpositive/groupby_sort_2.q.out b5e52f1 
  ql/src/test/results/clientpositive/groupby_sort_3.q.out c16911a 
  ql/src/test/results/clientpositive/groupby_sort_4.q.out a6b1c3d 
  ql/src/test/results/clientpositive/groupby_sort_5.q.out 369e2b5 
  ql/src/test/results/clientpositive/groupby_sort_7.q.out 7264695 
  ql/src/test/results/clientpositive/groupby_sort_8.q.out ec16eb0 
  ql/src/test/results/clientpositive/groupby_sort_9.q.out e49781a 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 0d631ce 
  ql/src/test/results/clientpositive/groupby_sort_test_1.q.out 8c1765d 
  ql/src/test/results/clientpositive/insert_orig_table.q.out 5eea74d 
  ql/src/test/results/clientpositive/insert_values_orig_table.q.out 684cd1b 
  ql/src/test/results/clientpositive/join_filters.q.out 4f112bd 
  ql/src/test/results/clientpositive/join_nulls.q.out 46e0170 
  ql/src/test/results/clientpositive/mergejoin.q.out cb96ab3 
  ql/src/test/results/clientpositive/skewjoin_mapjoin11.q.out 51445a5 
  ql/src/test/results/clientpositive/skewjoinopt19.q.out 91167db 
  ql/src/test/results/clientpositive/skewjoinopt20.q.out 15e96fd 
  ql/src/test/results/clientpositive/smb_mapjoin_1.q.out 9ab334b 
  ql/src/test/results/clientpositive/smb_mapjoin_10.q.out ea2fa51 
  ql/src/test/results/clientpositive/smb_mapjoin_2.q.out 379dc0d 
  ql/src/test/results/clientpositive/smb_mapjoin_25.q.out c0a8959 
  ql/src/test/results/clientpositive/smb_mapjoin_3.q.out 26fa5d4 
  ql/src/test/results/clientpositive/smb_mapjoin_4.q.out 9fc7f93 
  ql/src/test/results/clientpositive/smb_mapjoin_5.q.out 6e6882a 
  ql/src/test/results/clientpositive/smb_mapjoin_7.q.out 82f5804 
  ql/src/test/results/clientpositive/spark/auto_join32.q.out e26e4a2 
  ql/src/test/results/clientpositive/spark/auto_join_filters.q.out 8934433 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 09d2692 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out a70b161 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 2ea0a65 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out 6281929 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 31e9d86 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 3eceb0b 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out ddbca05 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 88d4dcb 
  ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out 7570ebe 
  ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out 80b44e9 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 65bded2 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 33e6d63 
  ql/src/test/results/clientpositive/spark/bucketmapjoin1.q.out d4a9c98 
  ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 678ad54 
  ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 95606f0 
  ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out d6c25e4 
  ql/src/test/results/clientpositive/spark/bucketmapjoin2.q.out d82480e 
  ql/src/test/results/clientpositive/spark/bucketmapjoin3.q.out 39552c1 
  ql/src/test/results/clientpositive/spark/bucketmapjoin4.q.out ad2762d 
  ql/src/test/results/clientpositive/spark/bucketmapjoin5.q.out f7c3d4d 
  ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out 7bfe440 
  ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 4601eb1 
  ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out 60bd103 
  ql/src/test/results/clientpositive/spark/bucketmapjoin_negative.q.out 031c46c 
  ql/src/test/results/clientpositive/spark/bucketmapjoin_negative2.q.out 4a8f46d 
  ql/src/test/results/clientpositive/spark/bucketmapjoin_negative3.q.out a09904e 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 65a8374 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 9f30e15 
  ql/src/test/results/clientpositive/spark/skewjoinopt19.q.out f51d805 
  ql/src/test/results/clientpositive/spark/skewjoinopt20.q.out 338da34 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_1.q.out 1ff1262 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_10.q.out cadf08e 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_2.q.out a0d51f3 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out cb811ed 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_3.q.out f46b833 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_4.q.out a421a42 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_5.q.out af65010 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out 622b950 
  ql/src/test/results/clientpositive/stats11.q.out e51f049 
  ql/src/test/results/clientpositive/tez/auto_join_filters.q.out 8fde41d 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out a275d27 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out 6ac74ca 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out e90af15 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out 2cb8416 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out abeceb8 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out 8eb9ce5 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_5.q.out adcc1fa 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out 2562cb0 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out 31b0a97 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 61c197f 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3f980b6 
  ql/src/test/results/clientpositive/tez/explainuser_2.q.out 0511819 
  ql/src/test/results/clientpositive/tez/insert_orig_table.q.out 5eea74d 
  ql/src/test/results/clientpositive/tez/mergejoin.q.out c4be404 
  ql/src/test/results/clientpositive/tez/tez_fsstat.q.out 3fcf68c 
  ql/src/test/results/clientpositive/tez/tez_smb_1.q.out d970bd9 
  ql/src/test/results/clientpositive/tez/tez_smb_main.q.out 52e1750 
  ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out 14a6874 

Diff: https://reviews.apache.org/r/34576/diff/


Testing
-------


Thanks,

pengcheng xiong


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message