hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
Date Fri, 26 Jul 2013 23:05:49 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721346#comment-13721346
] 

Hive QA commented on HIVE-4827:
-------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594435/HIVE-4827.4.patch

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/199/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh'
failed with exit status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-199/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'contrib/src/test/results/clientpositive/serde_typedbytes4.q.out'
Reverted 'contrib/src/test/results/clientpositive/dboutput.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join27.q.out'
Reverted 'ql/src/test/results/clientpositive/join14_hadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/udf9.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_clusterby.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join13.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_update.q.out'
Reverted 'ql/src/test/results/clientpositive/notable_alias1.q.out'
Reverted 'ql/src/test/results/clientpositive/load_dyn_part13.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_union.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_3.q.out'
Reverted 'ql/src/test/results/clientpositive/join38.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part7.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out'
Reverted 'ql/src/test/results/clientpositive/pcr.q.out'
Reverted 'ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/cluster.q.out'
Reverted 'ql/src/test/results/clientpositive/input12_hadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_file_format.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part2.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_transform.q.out'
Reverted 'ql/src/test/results/clientpositive/create_view.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_partition_coltype.q.out'
Reverted 'ql/src/test/results/clientpositive/infer_const_type.q.out'
Reverted 'ql/src/test/results/clientpositive/correlationoptimizer8.q.out'
Reverted 'ql/src/test/results/clientpositive/join7.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join4.q.out'
Reverted 'ql/src/test/results/clientpositive/input2_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/input11_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_like.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_hour.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_1.q.out'
Reverted 'ql/src/test/results/clientpositive/ppr_pushdown3.q.out'
Reverted 'ql/src/test/results/clientpositive/input11.q.out'
Reverted 'ql/src/test/results/clientpositive/noalias_subq1.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join29.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out'
Reverted 'ql/src/test/results/clientpositive/regex_col.q.out'
Reverted 'ql/src/test/results/clientpositive/allcolref_in_udf.q.out'
Reverted 'ql/src/test/results/clientpositive/udf1.q.out'
Reverted 'ql/src/test/results/clientpositive/join12.q.out'
Reverted 'ql/src/test/results/clientpositive/join35.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part4.q.out'
Reverted 'ql/src/test/results/clientpositive/union20.q.out'
Reverted 'ql/src/test/results/clientpositive/column_access_stats.q.out'
Reverted 'ql/src/test/results/clientpositive/join21.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_multi_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join6.q.out'
Reverted 'ql/src/test/results/clientpositive/correlationoptimizer10.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_lower.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup.q.out'
Reverted 'ql/src/test/results/clientpositive/join4.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_compression.q.out'
Reverted 'ql/src/test/results/clientpositive/input18.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert_gby.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_mult_tables.q.out'
Reverted 'ql/src/test/results/clientpositive/rand_partitionpruner3.q.out'
Reverted 'ql/src/test/results/clientpositive/input13.q.out'
Reverted 'ql/src/test/results/clientpositive/order2.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_udf_col.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join12.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_outer_join1.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_empty.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join21.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_2.q.out'
Reverted 'ql/src/test/results/clientpositive/union22.q.out'
Reverted 'ql/src/test/results/clientpositive/join23.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_second.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_multiple.q.out'
Reverted 'ql/src/test/results/clientpositive/union.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_10_trims.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part1.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join8.q.out'
Reverted 'ql/src/test/results/clientpositive/index_compression.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_parse_url.q.out'
Reverted 'ql/src/test/results/clientpositive/join41.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out'
Reverted 'ql/src/test/results/clientpositive/no_hooks.q.out'
Reverted 'ql/src/test/results/clientpositive/join6.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap3.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_position.q.out'
Reverted 'ql/src/test/results/clientpositive/transform_ppr2.q.out'
Reverted 'ql/src/test/results/clientpositive/set_processor_namespaces.q.out'
Reverted 'ql/src/test/results/clientpositive/union_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join14_hadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/nonmr_fetch.q.out'
Reverted 'ql/src/test/results/clientpositive/router_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join28.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_minute.q.out'
Reverted 'ql/src/test/results/clientpositive/input42.q.out'
Reverted 'ql/src/test/results/clientpositive/having.q.out'
Reverted 'ql/src/test/results/clientpositive/quote1.q.out'
Reverted 'ql/src/test/results/clientpositive/notable_alias2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_unused.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join23.q.out'
Reverted 'ql/src/test/results/clientpositive/join16.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_4.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup2.q.out'
Reverted 'ql/src/test/results/clientpositive/join39.q.out'
Reverted 'ql/src/test/results/clientpositive/union24.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto.q.out'
Reverted 'ql/src/test/results/clientpositive/rcfile_null_value.q.out'
Reverted 'ql/src/test/results/clientpositive/union33.q.out'
Reverted 'ql/src/test/results/clientpositive/join11.q.out'
Reverted 'ql/src/test/results/clientpositive/join34.q.out'
Reverted 'ql/src/test/results/clientpositive/join20.q.out'
Reverted 'ql/src/test/results/clientpositive/join8.q.out'
Reverted 'ql/src/test/results/clientpositive/subq.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join5.q.out'
Reverted 'ql/src/test/results/clientpositive/input14_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/louter_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join0.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_self_join.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join16.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup4.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join11.q.out'
Reverted 'ql/src/test/results/clientpositive/subq2.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join20.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out'
Reverted 'ql/src/test/results/clientpositive/join13.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part5.q.out'
Reverted 'ql/src/test/results/clientpositive/cast1.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join7.q.out'
Reverted 'ql/src/test/results/clientpositive/join40.q.out'
Reverted 'ql/src/test/results/clientpositive/join5.q.out'
Reverted 'ql/src/test/results/clientpositive/outer_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/input1_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto.q.out'
Reverted 'ql/src/test/results/clientpositive/transform_ppr1.q.out'
Reverted 'ql/src/test/results/clientpositive/join0.q.out'
Reverted 'ql/src/test/results/clientpositive/regexp_extract.q.out'
Reverted 'ql/src/test/results/clientpositive/ctas_colname.q.out'
Reverted 'ql/src/test/results/clientpositive/mergejoins.q.out'
Reverted 'ql/src/test/results/clientpositive/input14.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join27.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join13.q.out'
Reverted 'ql/src/test/results/beelinepositive/create_view.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join28.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join4.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join23.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join29.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join5.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join0.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join6.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join16.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join20.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join7.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join12.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join21.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join8.q.out'
Reverted 'ql/src/test/results/compiler/plan/input2.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input3.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join4.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input4.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join5.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join6.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join7.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join8.q.xml'
Reverted 'ql/src/test/results/compiler/plan/union.q.xml'
Reverted 'ql/src/test/results/compiler/plan/udf1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input_part1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/subq.q.xml'
Reverted 'ql/src/test/results/compiler/plan/cast1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input1.q.xml'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build
common/src/gen
+ svn update
D    testutils/ptest2/src/test/resources/TEST-SomeTest-failure.xml
D    testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin_union_remove_1.q-6fa31776-d2b0-4e13-9761-11f750627ad1-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D    testutils/ptest2/src/test/resources/test-outputs/TEST-union_remove_9.q-acb9de8f-1b9c-4874-924c-b2107ca7b07c-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D    testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin.q-ab8536a7-1b5c-45ed-ba29-14450f27db8b-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D    testutils/ptest2/src/test/resources/test-outputs/TEST-index_auth.q-bucketcontex-ba31fb54-1d7f-4c70-a89d-477b7d155191-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D    testutils/ptest2/src/test/resources/test-outputs/TEST-SomeTest-truncated.xml

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1507508.

Updated to revision 1507508.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}

This message is automatically generated.
                
> Merge a Map-only job to its following MapReduce job with multiple inputs
> ------------------------------------------------------------------------
>
>                 Key: HIVE-4827
>                 URL: https://issues.apache.org/jira/browse/HIVE-4827
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>    Affects Versions: 0.12.0
>            Reporter: Yin Huai
>            Assignee: Yin Huai
>         Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3.patch, HIVE-4827.4.patch
>
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a Map-only
job (MapJoin) to its following MapReduce job. But this merge only happens when the MapReduce
job has a single input. With Correlation Optimizer (HIVE-2206), it is possible that the MapReduce
job can have multiple inputs (for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
>       FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
>       GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
>       FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
>       GROUP BY x2.key2) tmp2
> ON (tmp1.key = tmp2.key)
> GROUP BY tmp1.key;
> {\code}
> In this query, join operations inside tmp1 and tmp2 will be converted to two MapJoins.
With Correlation Optimizer, aggregations in tmp1, tmp2, and join of tmp1 and tmp2, and the
last aggregation will be executed in the same MapReduce job (Reduce side). Since this MapReduce
job has two inputs, right now, CommonJoinResolver cannot attach two MapJoins to the Map side
of a MapReduce job.
> Another example:
> {code:sql}
> SELECT tmp1.key
> FROM (SELECT x1.key2 AS key
>       FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
>       UNION ALL
>       SELECT x2.key2 AS key
>       FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
> {\code}
> For this case, we will have three Map-only jobs (two for MapJoins and one for Union).
It will be good to use a single Map-only job to execute this query.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message