Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8A15710C3E for ; Sat, 18 Jan 2014 14:54:22 +0000 (UTC) Received: (qmail 87367 invoked by uid 500); 18 Jan 2014 14:54:20 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 87308 invoked by uid 500); 18 Jan 2014 14:54:20 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 87295 invoked by uid 99); 18 Jan 2014 14:54:19 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jan 2014 14:54:19 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 0EA951C05EC; Sat, 18 Jan 2014 14:54:19 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1370330163439019691==" MIME-Version: 1.0 Subject: Re: Review Request 15561: HIVE-5771 Constant propagation optimizer for Hive From: "Ted Xu" To: "Xuefu Zhang" , "hive" , "Ted Xu" Date: Sat, 18 Jan 2014 14:54:19 -0000 Message-ID: <20140118145419.31927.86836@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Ted Xu" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/15561/ X-Sender: "Ted Xu" References: <20131119184529.27204.96205@reviews.apache.org> In-Reply-To: <20131119184529.27204.96205@reviews.apache.org> Reply-To: "Ted Xu" X-ReviewRequest-Repository: hive --===============1370330163439019691== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit > On Nov. 19, 2013, 6:45 p.m., Xuefu Zhang wrote: > > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java, line 398 > > > > > > What is this fix for? Because ColumnPruner is applied after constant propagation optimizer(ConstantPO), some expressions are already eliminated by ConstantPO. This statement here is to prevent ArrayOutOfBoundException in that cases. - Ted ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15561/#review29119 ----------------------------------------------------------- On Jan. 18, 2014, 2:54 p.m., Ted Xu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15561/ > ----------------------------------------------------------- > > (Updated Jan. 18, 2014, 2:54 p.m.) > > > Review request for hive. > > > Bugs: HIVE-5771 > https://issues.apache.org/jira/browse/HIVE-5771 > > > Repository: hive > > > Description > ------- > > Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime. > > HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside. > > It may reduce I/O and accelerate process if we introduce such an optimizer. > > > Diffs > ----- > > http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1559345 > http://svn.apache.org/repos/asf/hive/trunk/contrib/src/test/results/clientpositive/udf_example_add.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/contrib/src/test/results/clientpositive/udf_example_format.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/alter_partition_coltype.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_filter.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_part.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_select.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog1.q PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog2.q PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog_dp.q PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog_type.q PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/decimal_udf.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/pcr.q 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cast1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/column_access_stats.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/constprog1.q.out PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/constprog2.q.out PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/constprog_dp.q.out PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/constprog_type.q.out PRE-CREATION > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby_ppd.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby_sort_1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_empty.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_file_format.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_multiple.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_partitioned.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_update.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_stale.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_stale_partitioned.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/infer_const_type.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input18.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input23.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input26.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input38.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input6.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input_part2.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input_part4.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/input_part6.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/insert1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_nullsafe.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_view.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/lateral_view_ppd.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/literal_double.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/macro.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/mapjoin1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/multi_insert.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/num_op_type_conv.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd2.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_constant_where.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_col.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_union_view.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/query_result_fileformat.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/quote1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/quotedid_basic.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/quotedid_partition.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/regexp_extract.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/sample8.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/select_unquote_and.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/select_unquote_not.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/select_unquote_or.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/set_processor_namespaces.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/set_variable_sub.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_18.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/stats_empty_dyn_part.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subq_where_serialization.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_exists.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_exists_having.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_in.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_in_having.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_multiinsert.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_notexists.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_notexists_having.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_notin.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/subquery_notin_having.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/transform_ppr1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/type_cast_1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/type_widening.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf1.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf4.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf5.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf6.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf7.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf9.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_10_trims.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_E.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_PI.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_abs.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_ascii.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_between.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_case.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_coalesce.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_degrees.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_elt.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_format_number.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_hash.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_hour.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_if.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_instr.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_isnull_isnotnull.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_like.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_locate.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_lower.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_lpad.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_minute.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_nvl.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_parse_url.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_printf.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_radians.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_reflect2.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_repeat.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_rpad.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_second.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_sign.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_space.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_when.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union33.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union_view.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/vectorization_div0.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/vectorization_short_regress.q.out 1559345 > http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/vectorized_math_funcs.q.out 1559345 > > Diff: https://reviews.apache.org/r/15561/diff/ > > > Testing > ------- > > > Thanks, > > Ted Xu > > --===============1370330163439019691==--