Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD93F11269 for ; Tue, 12 Aug 2014 13:21:14 +0000 (UTC) Received: (qmail 53470 invoked by uid 500); 12 Aug 2014 13:21:14 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 53405 invoked by uid 500); 12 Aug 2014 13:21:14 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 53388 invoked by uid 500); 12 Aug 2014 13:21:14 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 53385 invoked by uid 99); 12 Aug 2014 13:21:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Aug 2014 13:21:14 +0000 Date: Tue, 12 Aug 2014 13:21:14 +0000 (UTC) From: "Hive QA (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-7617) optimize bytes mapjoin hash table read path wrt serialization, at least for common cases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7617?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14094= 045#comment-14094045 ]=20 Hive QA commented on HIVE-7617: ------------------------------- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12661115/HIVE-7617.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job= /PreCommit-HIVE-TRUNK-Build/269/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/j= ob/PreCommit-HIVE-TRUNK-Build/269/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit= -HIVE-TRUNK-Build-269/ Messages: {noformat} **** This message was trimmed, see log for full details **** Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out' Reverted 'ql/src/test/results/clientpositive/windowing_udaf.q.out' Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_17.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_repeated_alias.q.out' Reverted 'ql/src/test/results/clientpositive/join36.q.out' Reverted 'ql/src/test/results/clientpositive/avro_joins.q.out' Reverted 'ql/src/test/results/clientpositive/drop_with_concurrency.q.out' Reverted 'ql/src/test/results/clientpositive/lock2.q.out' Reverted 'ql/src/test/results/clientpositive/input_part5.q.out' Reverted 'ql/src/test/results/clientpositive/bucketmapjoin3.q.out' Reverted 'ql/src/test/results/clientpositive/insert_into2.q.out' Reverted 'ql/src/test/results/clientpositive/alter_skewed_table.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out' Reverted 'ql/src/test/results/clientpositive/timestamp_1.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_insert.q.out' Reverted 'ql/src/test/results/clientpositive/partition_vs_table_metadata.q.= out' Reverted 'ql/src/test/results/clientpositive/groupby12.q.out' Reverted 'ql/src/test/results/clientpositive/alter_numbuckets_partitioned_t= able_h23.q.out' Reverted 'ql/src/test/results/clientpositive/infer_bucket_sort_list_bucket.= q.out' Reverted 'ql/src/test/results/clientpositive/groupby1_limit.q.out' Reverted 'ql/src/test/results/clientpositive/cast1.q.out' Reverted 'ql/src/test/results/clientpositive/input16_cc.q.out' Reverted 'ql/src/test/results/clientpositive/annotate_stats_part.q.out' Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat15.q.= out' Reverted 'ql/src/test/results/clientpositive/stats1.q.out' Reverted 'ql/src/test/results/clientpositive/symlink_text_input_format.q.ou= t' Reverted 'ql/src/test/results/clientpositive/ptf_register_tblfn.q.out' Reverted 'ql/src/test/results/clientpositive/ddltime.q.out' Reverted 'ql/src/test/results/clientpositive/uniquejoin.q.out' Reverted 'ql/src/test/results/clientpositive/truncate_column_list_bucket.q.= out' Reverted 'ql/src/test/results/clientpositive/ppd_constant_expr.q.out' Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_7.q.out' Reverted 'ql/src/test/results/clientpositive/union30.q.out' Reverted 'ql/src/test/results/clientpositive/orc_min_max.q.out' Reverted 'ql/src/test/results/clientpositive/order_within_subquery.q.out' Reverted 'ql/src/test/results/clientpositive/timestamp_null.q.out' Reverted 'ql/src/test/results/clientpositive/combine1.q.out' Reverted 'ql/src/test/results/clientpositive/avro_sanity_test.q.out' Reverted 'ql/src/test/results/clientpositive/exim_06_one_part.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_multi_insert_common_di= stinct.q.out' Reverted 'ql/src/test/results/clientpositive/groupby2_noskew.q.out' Reverted 'ql/src/test/results/clientpositive/index_stale.q.out' Reverted 'ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.o= ut' Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_12.q.out' Reverted 'ql/src/test/results/clientpositive/ptf_streaming.q.out' Reverted 'ql/src/test/results/clientpositive/orc_wide_table.q.out' Reverted 'ql/src/test/results/clientpositive/metadataonly1.q.out' Reverted 'ql/src/test/results/clientpositive/join31.q.out' Reverted 'ql/src/test/results/clientpositive/concatenate_inherit_table_loca= tion.q.out' Reverted 'ql/src/test/results/clientpositive/alter_rename_partition_authori= zation.q.out' Reverted 'ql/src/test/results/clientpositive/rcfile_toleratecorruptions.q.o= ut' Reverted 'ql/src/test/results/clientpositive/decimal_10_0.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join7.q.out' Reverted 'ql/src/test/results/clientpositive/nonreserved_keywords_insert_in= to1.q.out' Reverted 'ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.= out' Reverted 'ql/src/test/results/clientpositive/tablename_with_select.q.out' Reverted 'ql/src/test/results/clientpositive/exim_04_all_part.q.out' Reverted 'ql/src/test/results/clientpositive/join32_lessSize.q.out' Reverted 'ql/src/test/results/clientpositive/correlationoptimizer11.q.out' Reverted 'ql/src/test/results/clientpositive/multiMapJoin1.q.out' Reverted 'ql/src/test/results/clientpositive/ctas.q.out' Reverted 'ql/src/test/results/clientpositive/alter3.q.out' Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_21.q.out' Reverted 'ql/src/test/results/clientpositive/exim_23_import_part_authsucces= s.q.out' Reverted 'ql/src/test/results/clientpositive/groupby4_map_skew.q.out' Reverted 'ql/src/test/results/clientpositive/exim_01_nonpart.q.out' Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out' Reverted 'ql/src/test/results/clientpositive/groupby7_noskew.q.out' Reverted 'ql/src/test/results/clientpositive/test_boolean_whereclause.q.out= ' Reverted 'ql/src/test/results/clientpositive/drop_index_removes_partition_d= irs.q.out' Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat10.q.= out' Reverted 'ql/src/test/results/clientpositive/compute_stats_long.q.out' Reverted 'ql/src/test/results/clientpositive/inputddl7.q.out' Reverted 'ql/src/test/results/clientpositive/join5.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_5.q.out' Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_2.q.out' Reverted 'ql/src/test/results/clientpositive/describe_syntax.q.out' Reverted 'ql/src/test/results/clientpositive/add_part_multiple.q.out' Reverted 'ql/src/test/results/clientpositive/external_table_with_space_in_l= ocation_path.q.out' Reverted 'ql/src/test/results/clientpositive/fileformat_text.q.out' Reverted 'ql/src/test/results/clientpositive/char_udf1.q.out' Reverted 'ql/src/test/results/clientpositive/date_1.q.out' Reverted 'ql/src/test/results/clientpositive/exim_08_nonpart_rename.q.out' Reverted 'ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.ou= t' Reverted 'ql/src/test/results/clientpositive/udf_in_file.q.out' Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_10.q.out' Reverted 'ql/src/test/results/clientpositive/windowing.q.out' Reverted 'ql/src/test/results/clientpositive/bucket_if_with_path_filter.q.o= ut' Reverted 'ql/src/test/results/clientpositive/auto_join2.q.out' Reverted 'ql/src/test/results/clientpositive/udf_reverse.q.out' Reverted 'ql/src/test/results/clientpositive/union_remove_15.q.out' Reverted 'ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.= out' Reverted 'ql/src/test/results/clientpositive/union3.q.out' Reverted 'ql/src/test/results/clientpositive/show_create_table_partitioned.= q.out' Reverted 'ql/src/test/results/clientpositive/input1_limit.q.out' Reverted 'ql/src/test/results/clientpositive/groupby8.q.out' Reverted 'ql/src/test/results/clientpositive/scriptfile1.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap2.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_sort_9.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_grouping_id1.q.out' Reverted 'ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out' Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out' Reverted 'ql/src/test/results/clientpositive/alter_varchar2.q.out' Reverted 'ql/src/test/results/clientpositive/skewjoinopt6.q.out' Reverted 'ql/src/test/results/clientpositive/load_dyn_part6.q.out' Reverted 'ql/src/test/results/clientpositive/inputddl2.q.out' Reverted 'ql/src/test/results/clientpositive/drop_partitions_filter2.q.out' Reverted 'ql/src/test/results/clientpositive/sample7.q.out' Reverted 'ql/src/test/results/clientpositive/union_remove_24.q.out' Reverted 'ql/src/test/results/clientpositive/rcfile_union.q.out' Reverted 'ql/src/test/results/clientpositive/nullinput2.q.out' Reverted 'ql/src/test/results/clientpositive/temp_table.q.out' Reverted 'ql/src/test/results/clientpositive/exim_10_external_managed.q.out= ' Reverted 'ql/src/test/results/clientpositive/ptf_general_queries.q.out' Reverted 'ql/src/test/results/clientpositive/create_escape.q.out' Reverted 'ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverl= ap.q.out' Reverted 'ql/src/test/results/clientpositive/annotate_stats_limit.q.out' Reverted 'ql/src/test/results/clientpositive/alter_char2.q.out' Reverted 'ql/src/test/results/clientpositive/input_testsequencefile.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_join5.q.out' Reverted 'ql/src/test/results/clientpositive/join_view.q.out' Reverted 'ql/src/test/results/clientpositive/alter_merge_orc.q.out' Reverted 'ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization= .q.out' Reverted 'ql/src/test/results/clientpositive/udf_get_json_object.q.out' Reverted 'ql/src/test/results/clientpositive/input_testxpath.q.out' Reverted 'ql/src/test/results/clientpositive/union_remove_10.q.out' Reverted 'ql/src/test/results/clientpositive/input19.q.out' Reverted 'ql/src/test/results/clientpositive/bucket_map_join_2.q.out' Reverted 'ql/src/test/results/clientpositive/groupby3.q.out' Reverted 'ql/src/test/results/clientpositive/ba_table3.q.out' Reverted 'ql/src/test/results/clientpositive/join_empty.q.out' Reverted 'ql/src/test/results/clientpositive/windowing_rank.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_explain.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_sort_4.q.out' Reverted 'ql/src/test/results/clientpositive/parquet_partitioned.q.out' Reverted 'ql/src/test/results/clientpositive/reduce_deduplicate_exclude_gby= .q.out' Reverted 'ql/src/test/results/clientpositive/skewjoinopt18.q.out' Reverted 'ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out' Reverted 'ql/src/test/results/clientpositive/skewjoinopt1.q.out' Reverted 'ql/src/test/results/clientpositive/load_dyn_part1.q.out' Reverted 'ql/src/test/results/clientpositive/sample2.q.out' Reverted 'ql/src/test/results/clientpositive/exim_03_nonpart_over_compat.q.= out' Reverted 'ql/src/test/results/clientpositive/stats16.q.out' Reverted 'ql/src/test/results/clientpositive/temp_table_gb1.q.out' Reverted 'ql/src/test/results/clientpositive/input28.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_grant_public_rol= e.q.out' Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_4.q.out' Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out' Reverted 'ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.= out' Reverted 'ql/src/test/results/clientpositive/udf_using.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_union_view.q.out' Reverted 'ql/src/test/results/clientpositive/input_testxpath2.q.out' Reverted 'ql/src/test/results/clientpositive/create_like_tbl_props.q.out' Reverted 'ql/src/test/results/clientpositive/groupby7_map_multi_single_redu= cer.q.out' Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat9.q.o= ut' Reverted 'ql/src/test/results/clientpositive/truncate_column.q.out' Reverted 'ql/src/test/results/clientpositive/mergejoins.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_complex_types.q.out' Reverted 'ql/src/test/results/clientpositive/input14.q.out' Reverted 'ql/src/test/results/clientpositive/bucketcontext_5.q.out' Reverted 'ql/src/test/results/clientpositive/input37.q.out' Reverted 'ql/src/test/results/clientpositive/binary_table_colserde.q.out' Reverted 'ql/src/test/results/clientpositive/skewjoinopt13.q.out' Reverted 'ql/src/test/results/clientpositive/describe_comment_nonascii.q.ou= t' Reverted 'ql/src/test/results/clientpositive/orc_merge3.q.out' Reverted 'ql/src/test/results/clientpositive/mapreduce4.q.out' Reverted 'ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out' Reverted 'ql/src/test/results/clientpositive/union_remove_8.q.out' Reverted 'ql/src/test/results/clientpositive/nullscript.q.out' Reverted 'ql/src/test/results/clientpositive/stats11.q.out' Reverted 'ql/src/test/results/clientpositive/bucket5.q.out' Reverted 'ql/src/test/results/clientpositive/alter_merge_stats.q.out' Reverted 'ql/src/test/results/clientpositive/udf_round_2.q.out' Reverted 'ql/src/test/results/clientpositive/filter_numeric.q.out' Reverted 'ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out' Reverted 'ql/src/test/results/clientpositive/input46.q.out' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java= ' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.j= ava' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plug= in/sqlstd/Operation2Privilege.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plug= in/sqlstd/SQLStdHiveAuthorizationValidator.java' ++ awk '{print $2}' ++ svn status --no-ignore ++ egrep -v '^X|^Performing status on external' + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target s= hims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/ta= rget shims/common-secure/target packaging/target hbase-handler/target testu= tils/target jdbc/target metastore/target itests/target itests/hcatalog-unit= /target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoo= p2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-= serde/target itests/util/target hcatalog/target hcatalog/core/target hcatal= og/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig= -adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/ta= rget hwi/target common/target common/src/gen contrib/target service/target = serde/target beeline/target odbc/target cli/target ql/dependency-reduced-po= m.xml ql/target ql/src/test/results/clientnegative/authorization_drop_index= .q.out ql/src/test/results/clientnegative/authorization_create_index.q.out = ql/src/test/queries/clientnegative/authorization_create_index.q ql/src/test= /queries/clientnegative/authorization_drop_index.q + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1617468. At revision 1617468. + patchCommandPath=3D/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=3D/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/wo= rking/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12661115 > optimize bytes mapjoin hash table read path wrt serialization, at least f= or common cases > -------------------------------------------------------------------------= --------------- > > Key: HIVE-7617 > URL: https://issues.apache.org/jira/browse/HIVE-7617 > Project: Hive > Issue Type: Improvement > Reporter: Sergey Shelukhin > Assignee: Sergey Shelukhin > Attachments: HIVE-7617.patch, HIVE-7617.prelim.patch > > > BytesBytes has table stores keys in the byte array for compact representa= tion, however that means that the straightforward implementation of lookups= serializes lookup keys to byte arrays, which is relatively expensive. > We can either shortcut hashcode and compare for common types on read path= (integral types which would cover most of the real-world keys), or special= ize hashtable and from BytesBytes... create LongBytes, StringBytes, or what= ever. First one seems simpler now. -- This message was sent by Atlassian JIRA (v6.2#6252)