hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7617) optimize bytes mapjoin hash table read path wrt serialization, at least for common cases
Date Tue, 12 Aug 2014 13:21:14 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094045#comment-14094045
] 

Hive QA commented on HIVE-7617:
-------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12661115/HIVE-7617.patch

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/269/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/269/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-269/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out'
Reverted 'ql/src/test/results/clientpositive/windowing_udaf.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_17.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_repeated_alias.q.out'
Reverted 'ql/src/test/results/clientpositive/join36.q.out'
Reverted 'ql/src/test/results/clientpositive/avro_joins.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_with_concurrency.q.out'
Reverted 'ql/src/test/results/clientpositive/lock2.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part5.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketmapjoin3.q.out'
Reverted 'ql/src/test/results/clientpositive/insert_into2.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_skewed_table.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out'
Reverted 'ql/src/test/results/clientpositive/timestamp_1.q.out'
Reverted 'ql/src/test/results/clientpositive/authorization_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_vs_table_metadata.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby12.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out'
Reverted 'ql/src/test/results/clientpositive/infer_bucket_sort_list_bucket.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby1_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/cast1.q.out'
Reverted 'ql/src/test/results/clientpositive/input16_cc.q.out'
Reverted 'ql/src/test/results/clientpositive/annotate_stats_part.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat15.q.out'
Reverted 'ql/src/test/results/clientpositive/stats1.q.out'
Reverted 'ql/src/test/results/clientpositive/symlink_text_input_format.q.out'
Reverted 'ql/src/test/results/clientpositive/ptf_register_tblfn.q.out'
Reverted 'ql/src/test/results/clientpositive/ddltime.q.out'
Reverted 'ql/src/test/results/clientpositive/uniquejoin.q.out'
Reverted 'ql/src/test/results/clientpositive/truncate_column_list_bucket.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_constant_expr.q.out'
Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_7.q.out'
Reverted 'ql/src/test/results/clientpositive/union30.q.out'
Reverted 'ql/src/test/results/clientpositive/orc_min_max.q.out'
Reverted 'ql/src/test/results/clientpositive/order_within_subquery.q.out'
Reverted 'ql/src/test/results/clientpositive/timestamp_null.q.out'
Reverted 'ql/src/test/results/clientpositive/combine1.q.out'
Reverted 'ql/src/test/results/clientpositive/avro_sanity_test.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_06_one_part.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby2_noskew.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale.q.out'
Reverted 'ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_12.q.out'
Reverted 'ql/src/test/results/clientpositive/ptf_streaming.q.out'
Reverted 'ql/src/test/results/clientpositive/orc_wide_table.q.out'
Reverted 'ql/src/test/results/clientpositive/metadataonly1.q.out'
Reverted 'ql/src/test/results/clientpositive/join31.q.out'
Reverted 'ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out'
Reverted 'ql/src/test/results/clientpositive/rcfile_toleratecorruptions.q.out'
Reverted 'ql/src/test/results/clientpositive/decimal_10_0.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join7.q.out'
Reverted 'ql/src/test/results/clientpositive/nonreserved_keywords_insert_into1.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out'
Reverted 'ql/src/test/results/clientpositive/tablename_with_select.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_04_all_part.q.out'
Reverted 'ql/src/test/results/clientpositive/join32_lessSize.q.out'
Reverted 'ql/src/test/results/clientpositive/correlationoptimizer11.q.out'
Reverted 'ql/src/test/results/clientpositive/multiMapJoin1.q.out'
Reverted 'ql/src/test/results/clientpositive/ctas.q.out'
Reverted 'ql/src/test/results/clientpositive/alter3.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_21.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_23_import_part_authsuccess.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby4_map_skew.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_01_nonpart.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby7_noskew.q.out'
Reverted 'ql/src/test/results/clientpositive/test_boolean_whereclause.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_index_removes_partition_dirs.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat10.q.out'
Reverted 'ql/src/test/results/clientpositive/compute_stats_long.q.out'
Reverted 'ql/src/test/results/clientpositive/inputddl7.q.out'
Reverted 'ql/src/test/results/clientpositive/join5.q.out'
Reverted 'ql/src/test/results/clientpositive/authorization_5.q.out'
Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_2.q.out'
Reverted 'ql/src/test/results/clientpositive/describe_syntax.q.out'
Reverted 'ql/src/test/results/clientpositive/add_part_multiple.q.out'
Reverted 'ql/src/test/results/clientpositive/external_table_with_space_in_location_path.q.out'
Reverted 'ql/src/test/results/clientpositive/fileformat_text.q.out'
Reverted 'ql/src/test/results/clientpositive/char_udf1.q.out'
Reverted 'ql/src/test/results/clientpositive/date_1.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_08_nonpart_rename.q.out'
Reverted 'ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_in_file.q.out'
Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_10.q.out'
Reverted 'ql/src/test/results/clientpositive/windowing.q.out'
Reverted 'ql/src/test/results/clientpositive/bucket_if_with_path_filter.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join2.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_reverse.q.out'
Reverted 'ql/src/test/results/clientpositive/union_remove_15.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out'
Reverted 'ql/src/test/results/clientpositive/union3.q.out'
Reverted 'ql/src/test/results/clientpositive/show_create_table_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/input1_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby8.q.out'
Reverted 'ql/src/test/results/clientpositive/scriptfile1.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap2.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_9.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_grouping_id1.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_varchar2.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoinopt6.q.out'
Reverted 'ql/src/test/results/clientpositive/load_dyn_part6.q.out'
Reverted 'ql/src/test/results/clientpositive/inputddl2.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_partitions_filter2.q.out'
Reverted 'ql/src/test/results/clientpositive/sample7.q.out'
Reverted 'ql/src/test/results/clientpositive/union_remove_24.q.out'
Reverted 'ql/src/test/results/clientpositive/rcfile_union.q.out'
Reverted 'ql/src/test/results/clientpositive/nullinput2.q.out'
Reverted 'ql/src/test/results/clientpositive/temp_table.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_10_external_managed.q.out'
Reverted 'ql/src/test/results/clientpositive/ptf_general_queries.q.out'
Reverted 'ql/src/test/results/clientpositive/create_escape.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out'
Reverted 'ql/src/test/results/clientpositive/annotate_stats_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_char2.q.out'
Reverted 'ql/src/test/results/clientpositive/input_testsequencefile.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_join5.q.out'
Reverted 'ql/src/test/results/clientpositive/join_view.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_merge_orc.q.out'
Reverted 'ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_get_json_object.q.out'
Reverted 'ql/src/test/results/clientpositive/input_testxpath.q.out'
Reverted 'ql/src/test/results/clientpositive/union_remove_10.q.out'
Reverted 'ql/src/test/results/clientpositive/input19.q.out'
Reverted 'ql/src/test/results/clientpositive/bucket_map_join_2.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby3.q.out'
Reverted 'ql/src/test/results/clientpositive/ba_table3.q.out'
Reverted 'ql/src/test/results/clientpositive/join_empty.q.out'
Reverted 'ql/src/test/results/clientpositive/windowing_rank.q.out'
Reverted 'ql/src/test/results/clientpositive/authorization_explain.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_4.q.out'
Reverted 'ql/src/test/results/clientpositive/parquet_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/reduce_deduplicate_exclude_gby.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoinopt18.q.out'
Reverted 'ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoinopt1.q.out'
Reverted 'ql/src/test/results/clientpositive/load_dyn_part1.q.out'
Reverted 'ql/src/test/results/clientpositive/sample2.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_03_nonpart_over_compat.q.out'
Reverted 'ql/src/test/results/clientpositive/stats16.q.out'
Reverted 'ql/src/test/results/clientpositive/temp_table_gb1.q.out'
Reverted 'ql/src/test/results/clientpositive/input28.q.out'
Reverted 'ql/src/test/results/clientpositive/authorization_grant_public_role.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_4.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out'
Reverted 'ql/src/test/results/clientpositive/disable_merge_for_bucketing.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_using.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_union_view.q.out'
Reverted 'ql/src/test/results/clientpositive/input_testxpath2.q.out'
Reverted 'ql/src/test/results/clientpositive/create_like_tbl_props.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby7_map_multi_single_reducer.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out'
Reverted 'ql/src/test/results/clientpositive/truncate_column.q.out'
Reverted 'ql/src/test/results/clientpositive/mergejoins.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_complex_types.q.out'
Reverted 'ql/src/test/results/clientpositive/input14.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketcontext_5.q.out'
Reverted 'ql/src/test/results/clientpositive/input37.q.out'
Reverted 'ql/src/test/results/clientpositive/binary_table_colserde.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoinopt13.q.out'
Reverted 'ql/src/test/results/clientpositive/describe_comment_nonascii.q.out'
Reverted 'ql/src/test/results/clientpositive/orc_merge3.q.out'
Reverted 'ql/src/test/results/clientpositive/mapreduce4.q.out'
Reverted 'ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out'
Reverted 'ql/src/test/results/clientpositive/union_remove_8.q.out'
Reverted 'ql/src/test/results/clientpositive/nullscript.q.out'
Reverted 'ql/src/test/results/clientpositive/stats11.q.out'
Reverted 'ql/src/test/results/clientpositive/bucket5.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_merge_stats.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_round_2.q.out'
Reverted 'ql/src/test/results/clientpositive/filter_numeric.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out'
Reverted 'ql/src/test/results/clientpositive/input46.q.out'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java'
++ awk '{print $2}'
++ svn status --no-ignore
++ egrep -v '^X|^Performing status on external'
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target
hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen
contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml
ql/target ql/src/test/results/clientnegative/authorization_drop_index.q.out ql/src/test/results/clientnegative/authorization_create_index.q.out
ql/src/test/queries/clientnegative/authorization_create_index.q ql/src/test/queries/clientnegative/authorization_drop_index.q
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1617468.

At revision 1617468.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12661115

> optimize bytes mapjoin hash table read path wrt serialization, at least for common cases
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-7617
>                 URL: https://issues.apache.org/jira/browse/HIVE-7617
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-7617.patch, HIVE-7617.prelim.patch
>
>
> BytesBytes has table stores keys in the byte array for compact representation, however
that means that the straightforward implementation of lookups serializes lookup keys to byte
arrays, which is relatively expensive.
> We can either shortcut hashcode and compare for common types on read path (integral types
which would cover most of the real-world keys), or specialize hashtable and from BytesBytes...
create LongBytes, StringBytes, or whatever. First one seems simpler now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message