hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pengcheng xiong" <pxi...@hortonworks.com>
Subject Review Request 34576: Bucketized Table feature fails in some cases
Date Thu, 21 May 2015 23:34:50 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34576/
-----------------------------------------------------------

Review request for hive and John Pullokkaran.


Repository: hive-git


Description
-------

Bucketized Table feature fails in some cases. if src & destination is bucketed on same
key, and if actual data in the src is not bucketed (because data got loaded using LOAD DATA
LOCAL INPATH ) then the data won't be bucketed while writing to destination.
Example
----------------------------------------------------------------------
CREATE TABLE P1(key STRING, val STRING)
CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH '/Users/jp/apache-hive1/data/files/P1.txt' INTO TABLE P1;
– perform an insert to make sure there are 2 files
INSERT OVERWRITE TABLE P1 select key, val from P1;
--------------------------------------------------
This is not a regression. This has never worked.
This got only discovered due to Hadoop2 changes.
In Hadoop1, in local mode, number of reducers will always be 1, regardless of what is requested
by app. Hadoop2 now honors the number of reducer setting in local mode (by spawning threads).
Long term solution seems to be to prevent load data for bucketed table.


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java e53933e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 1a9b42b 
  ql/src/test/results/clientnegative/alter_partition_invalidspec.q.out 404115f 
  ql/src/test/results/clientnegative/alter_partition_nodrop.q.out 1c78cff 
  ql/src/test/results/clientnegative/alter_partition_nodrop_table.q.out 3c425da 
  ql/src/test/results/clientnegative/alter_partition_offline.q.out c70fcb4 
  ql/src/test/results/clientnegative/archive_corrupt.q.out 56e8ec4 
  ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
  ql/src/test/results/clientnegative/bucket_mapjoin_wrong_table_metadata_2.q.out 9aa9b5d 
  ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out 4ea70e3

  ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out ce79830

  ql/src/test/results/clientnegative/dynamic_partitions_with_whitelist.q.out f069ae8 
  ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 3c05600 
  ql/src/test/results/clientnegative/exim_15_part_nonpart.q.out dfbf025 
  ql/src/test/results/clientnegative/exim_16_part_noncompat_schema.q.out 4cb6ca7 
  ql/src/test/results/clientnegative/exim_17_part_spec_underspec.q.out 23caa4a 
  ql/src/test/results/clientnegative/exim_18_part_spec_missing.q.out 23caa4a 
  ql/src/test/results/clientnegative/exim_21_part_managed_external.q.out fd27f29 
  ql/src/test/results/clientnegative/exim_24_import_part_authfail.q.out 1a9a34d 
  ql/src/test/results/clientnegative/insertover_dynapart_ifnotexists.q.out a40ffab 
  ql/src/test/results/clientnegative/load_exist_part_authfail.q.out 491cfd0 
  ql/src/test/results/clientnegative/load_part_authfail.q.out 4ea8be9 
  ql/src/test/results/clientnegative/load_part_nospec.q.out bebaf92 
  ql/src/test/results/clientnegative/nopart_load.q.out 8815146 
  ql/src/test/results/clientnegative/protectmode_part2.q.out 16d58c7 
  ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out ffcbcf9 
  ql/src/test/results/clientpositive/alter_merge.q.out 17d86b8 
  ql/src/test/results/clientpositive/alter_merge_2.q.out e118c39 
  ql/src/test/results/clientpositive/alter_merge_stats.q.out fdd2ddc 
  ql/src/test/results/clientpositive/alter_partition_protect_mode.q.out 80990d9 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a2 
  ql/src/test/results/clientpositive/alter_table_cascade.q.out 0139466 
  ql/src/test/results/clientpositive/auto_join32.q.out bfc8be8 
  ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 383defd 
  ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out e6e7ef3 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out e9fb705 
  ql/src/test/results/clientpositive/auto_sortmerge_join_16.q.out d4ecb19 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c089419 
  ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 6e443fa 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out feaea04 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e89f548 
  ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 44c037f 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 870ecdd 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 33f5c46 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out 067d1ff 
  ql/src/test/results/clientpositive/bucketcontext_1.q.out 77bfcf9 
  ql/src/test/results/clientpositive/bucketcontext_2.q.out a9db13d 
  ql/src/test/results/clientpositive/bucketcontext_3.q.out 9ba3e0c 
  ql/src/test/results/clientpositive/bucketcontext_4.q.out a2b37a8 
  ql/src/test/results/clientpositive/bucketcontext_6.q.out d2304fa 
  ql/src/test/results/clientpositive/bucketcontext_7.q.out 1a105ed 
  ql/src/test/results/clientpositive/bucketcontext_8.q.out 138e415 
  ql/src/test/results/clientpositive/bucketizedhiveinputformat_auto.q.out 215efdd 
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out 72f2a07 
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out b0e849d 
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out 4263cab 
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out bcd7394 
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out a8d9e9d 
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out c759f05 
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out f61500c 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out 0cb2825 
  ql/src/test/results/clientpositive/bucketmapjoin7.q.out 667a9db 
  ql/src/test/results/clientpositive/bucketmapjoin8.q.out 252b377 
  ql/src/test/results/clientpositive/bucketmapjoin9.q.out 5e28dc3 
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 6ae127d 
  ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 4c9f54a 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out 3c22d40 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 18a6909 
  ql/src/test/results/clientpositive/database.q.out 043d91b 
  ql/src/test/results/clientpositive/database_drop.q.out 225104f 
  ql/src/test/results/clientpositive/drop_partition_with_stats.q.out e27e557 
  ql/src/test/results/clientpositive/exim_02_part.q.out 6e0988a 
  ql/src/test/results/clientpositive/exim_04_all_part.q.out 862efa3 
  ql/src/test/results/clientpositive/exim_05_some_part.q.out 1b6a515 
  ql/src/test/results/clientpositive/exim_06_one_part.q.out 39c83c3 
  ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out b55a0bd 
  ql/src/test/results/clientpositive/exim_08_nonpart_rename.q.out 740833b 
  ql/src/test/results/clientpositive/exim_09_part_spec_nonoverlap.q.out d71f36f 
  ql/src/test/results/clientpositive/exim_15_external_part.q.out d24f18a 
  ql/src/test/results/clientpositive/exim_16_part_external.q.out af748c9 
  ql/src/test/results/clientpositive/exim_17_part_managed.q.out a92f95a 
  ql/src/test/results/clientpositive/exim_18_part_external.q.out a082a11 
  ql/src/test/results/clientpositive/exim_19_00_part_external_location.q.out 5a97e03 
  ql/src/test/results/clientpositive/exim_19_part_external_location.q.out f9a20f7 
  ql/src/test/results/clientpositive/exim_20_part_managed_location.q.out b196ba5 
  ql/src/test/results/clientpositive/exim_23_import_part_authsuccess.q.out 5f78a76 
  ql/src/test/results/clientpositive/exim_hidden_files.q.out e449e0e 
  ql/src/test/results/clientpositive/global_limit.q.out 7da20d5 
  ql/src/test/results/clientpositive/groupby_sort_6.q.out c5cb8b9 
  ql/src/test/results/clientpositive/groupby_sort_7.q.out 7264695 
  ql/src/test/results/clientpositive/groupby_sort_8.q.out ec16eb0 
  ql/src/test/results/clientpositive/groupby_sort_9.q.out e49781a 
  ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 773a2a8 
  ql/src/test/results/clientpositive/input40.q.out bb0eabe 
  ql/src/test/results/clientpositive/inputddl6.q.out 5a040e6 
  ql/src/test/results/clientpositive/inputddl7.q.out 0d64baf 
  ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out 900babe 
  ql/src/test/results/clientpositive/insert2_overwrite_partitions.q.out 25c438f 
  ql/src/test/results/clientpositive/leftsemijoin.q.out 11f0bb0 
  ql/src/test/results/clientpositive/load_exist_part_authsuccess.q.out 8ec7e62 
  ql/src/test/results/clientpositive/load_part_authsuccess.q.out 8249dce 
  ql/src/test/results/clientpositive/loadpart2.q.out 201a957 
  ql/src/test/results/clientpositive/merge_dynamic_partition.q.out da19b32 
  ql/src/test/results/clientpositive/merge_dynamic_partition2.q.out 5a2afb0 
  ql/src/test/results/clientpositive/merge_dynamic_partition3.q.out 86978f3 
  ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 86af660 
  ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out c1468c1 
  ql/src/test/results/clientpositive/mergejoin.q.out cb96ab3 
  ql/src/test/results/clientpositive/nullgroup3.q.out 7712d4d 
  ql/src/test/results/clientpositive/nullgroup5.q.out 8a94d62 
  ql/src/test/results/clientpositive/orc_analyze.q.out a61a2e6 
  ql/src/test/results/clientpositive/orc_split_elimination.q.out 7134ff5 
  ql/src/test/results/clientpositive/parquet_serde.q.out e753180 
  ql/src/test/results/clientpositive/partition_type_check.q.out e25d527 
  ql/src/test/results/clientpositive/partition_wise_fileformat17.q.out 028a26e 
  ql/src/test/results/clientpositive/partition_wise_fileformat18.q.out 6303d44 
  ql/src/test/results/clientpositive/repl_1_drop.q.out 9fb65d1 
  ql/src/test/results/clientpositive/repl_2_exim_basic.q.out 8df0653 
  ql/src/test/results/clientpositive/repl_3_exim_metadata.q.out 8387c02 
  ql/src/test/results/clientpositive/smb_mapjoin_10.q.out ea2fa51 
  ql/src/test/results/clientpositive/spark/auto_join32.q.out 361a968 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 09d2692 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out 8102ec1 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_16.q.out d4ecb19 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out 2ea0a65 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out 6281929 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out 31e9d86 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out ddbca05 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out 88d4dcb 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 65bded2 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 33e6d63 
  ql/src/test/results/clientpositive/spark/bucketmapjoin1.q.out 44f4d0c 
  ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out 678ad54 
  ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out 95606f0 
  ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out d6c25e4 
  ql/src/test/results/clientpositive/spark/bucketmapjoin2.q.out d82480e 
  ql/src/test/results/clientpositive/spark/bucketmapjoin3.q.out 39552c1 
  ql/src/test/results/clientpositive/spark/bucketmapjoin4.q.out ad2762d 
  ql/src/test/results/clientpositive/spark/bucketmapjoin5.q.out f7c3d4d 
  ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out 7bfe440 
  ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out 4601eb1 
  ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out 60bd103 
  ql/src/test/results/clientpositive/spark/bucketmapjoin_negative.q.out 031c46c 
  ql/src/test/results/clientpositive/spark/bucketmapjoin_negative2.q.out 4a8f46d 
  ql/src/test/results/clientpositive/spark/leftsemijoin.q.out 11f0bb0 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_10.q.out cadf08e 
  ql/src/test/results/clientpositive/spark/stats18.q.out a061846 
  ql/src/test/results/clientpositive/spark/stats_counter_partitioned.q.out 4b84eca 
  ql/src/test/results/clientpositive/spark/statsfs.q.out b0bca41 
  ql/src/test/results/clientpositive/stats11.q.out e51f049 
  ql/src/test/results/clientpositive/stats18.q.out a061846 
  ql/src/test/results/clientpositive/stats_counter_partitioned.q.out ab1270c 
  ql/src/test/results/clientpositive/statsfs.q.out b0bca41 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out a275d27 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out 6ac74ca 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out 8c8a3bf 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_16.q.out d4ecb19 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out 2cb8416 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out abeceb8 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out 8eb9ce5 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out 2562cb0 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out 31b0a97 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 61c197f 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3f980b6 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 8b0b81d 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out b684858 
  ql/src/test/results/clientpositive/tez/explainuser_2.q.out f84524b 
  ql/src/test/results/clientpositive/tez/leftsemijoin.q.out 11f0bb0 
  ql/src/test/results/clientpositive/tez/mergejoin.q.out 97df12a 
  ql/src/test/results/clientpositive/tez/orc_analyze.q.out a61a2e6 
  ql/src/test/results/clientpositive/tez/stats_counter_partitioned.q.out ab1270c 
  ql/src/test/results/clientpositive/tez/tez_fsstat.q.out 3fcf68c 
  ql/src/test/results/clientpositive/tez/tez_smb_1.q.out d970bd9 
  ql/src/test/results/clientpositive/tez/tez_smb_main.q.out 6183390 
  ql/src/test/results/clientpositive/truncate_table.q.out 4d8f38c 
  ql/src/test/results/clientpositive/view_cast.q.out 34444ae 

Diff: https://reviews.apache.org/r/34576/diff/


Testing
-------


Thanks,

pengcheng xiong


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message