Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 50093200BA1 for ; Mon, 17 Oct 2016 22:42:00 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 4E99D160AEC; Mon, 17 Oct 2016 20:42:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CD35D160AFC for ; Mon, 17 Oct 2016 22:41:57 +0200 (CEST) Received: (qmail 38240 invoked by uid 500); 17 Oct 2016 20:41:54 -0000 Mailing-List: contact commits-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hive.apache.org Delivered-To: mailing list commits@hive.apache.org Received: (qmail 37539 invoked by uid 99); 17 Oct 2016 20:41:54 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Oct 2016 20:41:54 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 32635E098D; Mon, 17 Oct 2016 20:41:54 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sershe@apache.org To: commits@hive.apache.org Date: Mon, 17 Oct 2016 20:42:08 -0000 Message-Id: <03eb2829cece4c71a5211eeebddd1714@git.apache.org> In-Reply-To: <25f5e926c22c40baadb700cf217f2b65@git.apache.org> References: <25f5e926c22c40baadb700cf217f2b65@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [16/67] [abbrv] [partial] hive git commit: Revert "Revert "HIVE-11394: Enhance EXPLAIN display for vectorization (Matt McCline, reviewed by Gopal Vijayaraghavan)"" archived-at: Mon, 17 Oct 2016 20:42:00 -0000 http://git-wip-us.apache.org/repos/asf/hive/blob/16d28b34/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out index 9eeb0d6..26fa9d9 100644 --- a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out +++ b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out @@ -256,12 +256,16 @@ POSTHOOK: Input: default@flights_tiny_orc 2010-10-29 12 2010-10-30 11 2010-10-31 8 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select * from flights_tiny_orc sort by fl_num, fl_date limit 25 PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select * from flights_tiny_orc sort by fl_num, fl_date limit 25 POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -280,46 +284,102 @@ STAGE PLANS: TableScan alias: flights_tiny_orc Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col5 (type: int), _col2 (type: date) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col3 (type: timestamp), _col4 (type: float) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY.reducesinkkey1 (type: date), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 1, 4, 5, 0] Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col5 (type: int), _col2 (type: date) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col3 (type: timestamp), _col4 (type: float) Reducer 3 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY.reducesinkkey1 (type: date), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 1, 4, 5, 0] Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -365,12 +425,16 @@ Chicago New York 2010-10-24 2010-10-24 07:00:00 113.0 897 Chicago New York 2010-10-25 2010-10-25 07:00:00 -1.0 897 Chicago New York 2010-10-26 2010-10-26 07:00:00 0.0 897 Chicago New York 2010-10-27 2010-10-27 07:00:00 -11.0 897 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select fl_date, count(*) from flights_tiny_orc group by fl_date PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select fl_date, count(*) from flights_tiny_orc group by fl_date POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -388,12 +452,26 @@ STAGE PLANS: TableScan alias: flights_tiny_orc Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: fl_date (type: date) outputColumnNames: fl_date + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2] Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() + Group By Vectorization: + aggregators: VectorUDAFCountStar(*) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 2 + native: false + projectedOutputColumns: [0] keys: fl_date (type: date) mode: hash outputColumnNames: _col0, _col1 @@ -402,21 +480,50 @@ STAGE PLANS: key expressions: _col0 (type: date) sort order: + Map-reduce partition columns: _col0 (type: date) + Reduce Sink Vectorization: + className: VectorReduceSinkMultiKeyOperator + native: true + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: bigint) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) + Group By Vectorization: + aggregators: VectorUDAFCountMerge(col 1) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 0 + native: false + projectedOutputColumns: [0] keys: KEY._col0 (type: date) mode: mergepartial outputColumnNames: _col0, _col1 Statistics: Num rows: 68 Data size: 19584 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false Statistics: Num rows: 68 Data size: 19584 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -590,17 +697,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-29 POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-30 POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31 #### A masked pattern was here #### -Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20 -Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20 -Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20 -Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20 -Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20 -Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20 -Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20 -Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20 -Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20 -Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20 -Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20 +Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26 +Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26 +Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26 +Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26 +Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26 +Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26 +Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26 +Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26 +Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26 Baltimore New York 2010-10-21 07:00:00 17.0 1064 2010-10-21 Baltimore New York 2010-10-21 07:00:00 105.0 1142 2010-10-21 Baltimore New York 2010-10-21 07:00:00 28.0 1599 2010-10-21 @@ -613,41 +722,6 @@ Cleveland New York 2010-10-21 07:00:00 3.0 2630 2010-10-21 Cleveland New York 2010-10-21 07:00:00 29.0 2646 2010-10-21 Cleveland New York 2010-10-21 07:00:00 72.0 3014 2010-10-21 Washington New York 2010-10-21 07:00:00 22.0 7291 2010-10-21 -Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22 -Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22 -Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22 -Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22 -Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22 -Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22 -Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22 -Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22 -Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23 -Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23 -Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23 -Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23 -Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23 -Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23 -Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23 -Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23 -Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23 -Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23 -Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23 -Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23 -Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24 -Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24 -Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24 -Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24 -Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24 Baltimore New York 2010-10-25 07:00:00 -25.0 1064 2010-10-25 Baltimore New York 2010-10-25 07:00:00 92.0 1142 2010-10-25 Baltimore New York 2010-10-25 07:00:00 106.0 1599 2010-10-25 @@ -660,19 +734,42 @@ Cleveland New York 2010-10-25 07:00:00 -4.0 2630 2010-10-25 Cleveland New York 2010-10-25 07:00:00 81.0 2646 2010-10-25 Cleveland New York 2010-10-25 07:00:00 42.0 3014 2010-10-25 Washington New York 2010-10-25 07:00:00 9.0 7291 2010-10-25 -Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26 -Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26 -Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26 -Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26 -Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26 -Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26 -Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26 -Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26 -Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26 +Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24 +Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24 +Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24 +Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24 +Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24 +Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23 +Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23 +Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23 +Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23 +Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23 +Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23 +Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23 +Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23 +Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23 +Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23 +Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23 +Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23 +Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29 +Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29 +Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29 +Chicago New York 2010-10-29 07:00:00 -12.0 361 2010-10-29 +Chicago New York 2010-10-29 07:00:00 -11.0 897 2010-10-29 +Chicago New York 2010-10-29 07:00:00 15.0 1531 2010-10-29 +Chicago New York 2010-10-29 07:00:00 -18.0 1610 2010-10-29 +Chicago New York 2010-10-29 07:00:00 -4.0 3198 2010-10-29 +Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29 +Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29 +Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29 +Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29 Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27 Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27 Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27 @@ -696,18 +793,6 @@ Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28 Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28 Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28 Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28 -Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29 -Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29 -Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29 -Chicago New York 2010-10-29 07:00:00 -12.0 361 2010-10-29 -Chicago New York 2010-10-29 07:00:00 -11.0 897 2010-10-29 -Chicago New York 2010-10-29 07:00:00 15.0 1531 2010-10-29 -Chicago New York 2010-10-29 07:00:00 -18.0 1610 2010-10-29 -Chicago New York 2010-10-29 07:00:00 -4.0 3198 2010-10-29 -Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29 -Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29 -Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29 -Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29 Baltimore New York 2010-10-30 07:00:00 14.0 272 2010-10-30 Baltimore New York 2010-10-30 07:00:00 -1.0 1805 2010-10-30 Baltimore New York 2010-10-30 07:00:00 5.0 3171 2010-10-30 @@ -719,6 +804,28 @@ Cleveland New York 2010-10-30 07:00:00 -23.0 2018 2010-10-30 Cleveland New York 2010-10-30 07:00:00 -12.0 2932 2010-10-30 Washington New York 2010-10-30 07:00:00 -27.0 5904 2010-10-30 Washington New York 2010-10-30 07:00:00 -16.0 5917 2010-10-30 +Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20 +Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20 +Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20 +Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20 +Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20 +Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20 +Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20 +Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20 +Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20 +Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20 +Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20 +Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22 +Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22 +Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22 +Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22 +Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22 +Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22 +Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22 +Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22 Baltimore New York 2010-10-31 07:00:00 -1.0 1599 2010-10-31 Baltimore New York 2010-10-31 07:00:00 -14.0 2571 2010-10-31 Chicago New York 2010-10-31 07:00:00 -25.0 361 2010-10-31 @@ -828,26 +935,67 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31 2010-10-29 12 2010-10-30 11 2010-10-31 8 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_date PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_date POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: - Stage-0 is a root stage + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 STAGE PLANS: + Stage: Stage-1 + Tez +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: flights_tiny_orc_partitioned_date + Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] + Select Operator + expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] + Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL + File Output Operator + compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false + Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Execution mode: vectorized, llap + LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true + Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: - TableScan - alias: flights_tiny_orc_partitioned_date - Select Operator - expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date) - outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - ListSink + ListSink PREHOOK: query: select * from flights_tiny_orc_partitioned_date PREHOOK: type: QUERY @@ -881,17 +1029,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-29 POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-30 POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31 #### A masked pattern was here #### -Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20 -Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20 -Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20 -Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20 -Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20 -Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20 -Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20 -Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20 -Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20 -Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20 -Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20 +Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26 +Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26 +Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26 +Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26 +Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26 +Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26 +Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26 +Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26 +Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26 +Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26 Baltimore New York 2010-10-21 07:00:00 17.0 1064 2010-10-21 Baltimore New York 2010-10-21 07:00:00 105.0 1142 2010-10-21 Baltimore New York 2010-10-21 07:00:00 28.0 1599 2010-10-21 @@ -904,41 +1054,6 @@ Cleveland New York 2010-10-21 07:00:00 3.0 2630 2010-10-21 Cleveland New York 2010-10-21 07:00:00 29.0 2646 2010-10-21 Cleveland New York 2010-10-21 07:00:00 72.0 3014 2010-10-21 Washington New York 2010-10-21 07:00:00 22.0 7291 2010-10-21 -Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22 -Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22 -Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22 -Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22 -Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22 -Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22 -Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22 -Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22 -Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22 -Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23 -Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23 -Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23 -Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23 -Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23 -Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23 -Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23 -Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23 -Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23 -Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23 -Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23 -Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23 -Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24 -Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24 -Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24 -Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24 -Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24 -Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24 -Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24 Baltimore New York 2010-10-25 07:00:00 -25.0 1064 2010-10-25 Baltimore New York 2010-10-25 07:00:00 92.0 1142 2010-10-25 Baltimore New York 2010-10-25 07:00:00 106.0 1599 2010-10-25 @@ -951,42 +1066,30 @@ Cleveland New York 2010-10-25 07:00:00 -4.0 2630 2010-10-25 Cleveland New York 2010-10-25 07:00:00 81.0 2646 2010-10-25 Cleveland New York 2010-10-25 07:00:00 42.0 3014 2010-10-25 Washington New York 2010-10-25 07:00:00 9.0 7291 2010-10-25 -Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26 -Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26 -Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26 -Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26 -Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26 -Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26 -Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26 -Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26 -Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26 -Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26 -Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27 -Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27 -Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27 -Chicago New York 2010-10-27 07:00:00 148.0 361 2010-10-27 -Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27 -Chicago New York 2010-10-27 07:00:00 70.0 1531 2010-10-27 -Chicago New York 2010-10-27 07:00:00 8.0 1610 2010-10-27 -Chicago New York 2010-10-27 07:00:00 21.0 3198 2010-10-27 -Cleveland New York 2010-10-27 07:00:00 16.0 2630 2010-10-27 -Cleveland New York 2010-10-27 07:00:00 27.0 3014 2010-10-27 -Washington New York 2010-10-27 07:00:00 26.0 7291 2010-10-27 -Baltimore New York 2010-10-28 07:00:00 -4.0 1064 2010-10-28 -Baltimore New York 2010-10-28 07:00:00 -14.0 1142 2010-10-28 -Baltimore New York 2010-10-28 07:00:00 -14.0 1599 2010-10-28 -Chicago New York 2010-10-28 07:00:00 2.0 361 2010-10-28 -Chicago New York 2010-10-28 07:00:00 2.0 897 2010-10-28 -Chicago New York 2010-10-28 07:00:00 -11.0 1531 2010-10-28 -Chicago New York 2010-10-28 07:00:00 3.0 1610 2010-10-28 -Chicago New York 2010-10-28 07:00:00 -18.0 3198 2010-10-28 -Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28 -Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28 -Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28 -Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28 +Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24 +Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24 +Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24 +Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24 +Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24 +Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24 +Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24 +Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23 +Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23 +Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23 +Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23 +Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23 +Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23 +Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23 +Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23 +Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23 +Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23 +Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23 +Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23 Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29 Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29 Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29 @@ -999,6 +1102,29 @@ Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29 Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29 Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29 Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29 +Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27 +Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27 +Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27 +Chicago New York 2010-10-27 07:00:00 148.0 361 2010-10-27 +Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27 +Chicago New York 2010-10-27 07:00:00 70.0 1531 2010-10-27 +Chicago New York 2010-10-27 07:00:00 8.0 1610 2010-10-27 +Chicago New York 2010-10-27 07:00:00 21.0 3198 2010-10-27 +Cleveland New York 2010-10-27 07:00:00 16.0 2630 2010-10-27 +Cleveland New York 2010-10-27 07:00:00 27.0 3014 2010-10-27 +Washington New York 2010-10-27 07:00:00 26.0 7291 2010-10-27 +Baltimore New York 2010-10-28 07:00:00 -4.0 1064 2010-10-28 +Baltimore New York 2010-10-28 07:00:00 -14.0 1142 2010-10-28 +Baltimore New York 2010-10-28 07:00:00 -14.0 1599 2010-10-28 +Chicago New York 2010-10-28 07:00:00 2.0 361 2010-10-28 +Chicago New York 2010-10-28 07:00:00 2.0 897 2010-10-28 +Chicago New York 2010-10-28 07:00:00 -11.0 1531 2010-10-28 +Chicago New York 2010-10-28 07:00:00 3.0 1610 2010-10-28 +Chicago New York 2010-10-28 07:00:00 -18.0 3198 2010-10-28 +Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28 +Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28 +Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28 +Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28 Baltimore New York 2010-10-30 07:00:00 14.0 272 2010-10-30 Baltimore New York 2010-10-30 07:00:00 -1.0 1805 2010-10-30 Baltimore New York 2010-10-30 07:00:00 5.0 3171 2010-10-30 @@ -1010,6 +1136,28 @@ Cleveland New York 2010-10-30 07:00:00 -23.0 2018 2010-10-30 Cleveland New York 2010-10-30 07:00:00 -12.0 2932 2010-10-30 Washington New York 2010-10-30 07:00:00 -27.0 5904 2010-10-30 Washington New York 2010-10-30 07:00:00 -16.0 5917 2010-10-30 +Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20 +Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20 +Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20 +Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20 +Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20 +Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20 +Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20 +Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20 +Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20 +Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20 +Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20 +Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22 +Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22 +Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22 +Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22 +Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22 +Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22 +Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22 +Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22 +Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22 Baltimore New York 2010-10-31 07:00:00 -1.0 1599 2010-10-31 Baltimore New York 2010-10-31 07:00:00 -14.0 2571 2010-10-31 Chicago New York 2010-10-31 07:00:00 -25.0 361 2010-10-31 @@ -1018,12 +1166,16 @@ Chicago New York 2010-10-31 07:00:00 -4.0 1531 2010-10-31 Chicago New York 2010-10-31 07:00:00 -22.0 1610 2010-10-31 Chicago New York 2010-10-31 07:00:00 -15.0 3198 2010-10-31 Washington New York 2010-10-31 07:00:00 -18.0 7282 2010-10-31 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_date sort by fl_num, fl_date limit 25 PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_date sort by fl_num, fl_date limit 25 POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -1042,46 +1194,102 @@ STAGE PLANS: TableScan alias: flights_tiny_orc_partitioned_date Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL Reduce Output Operator key expressions: _col4 (type: int), _col5 (type: date) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: timestamp), _col3 (type: float) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: date) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 4, 5, 0, 1] Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL Reduce Output Operator key expressions: _col4 (type: int), _col5 (type: date) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: timestamp), _col3 (type: float) Reducer 3 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: date) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 4, 5, 0, 1] Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL File Output Operator compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -1151,12 +1359,16 @@ Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24 Chicago New York 2010-10-25 07:00:00 -1.0 897 2010-10-25 Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26 Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select fl_date, count(*) from flights_tiny_orc_partitioned_date group by fl_date PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select fl_date, count(*) from flights_tiny_orc_partitioned_date group by fl_date POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -1174,12 +1386,26 @@ STAGE PLANS: TableScan alias: flights_tiny_orc_partitioned_date Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: fl_date (type: date) outputColumnNames: fl_date + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [5] Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator aggregations: count() + Group By Vectorization: + aggregators: VectorUDAFCountStar(*) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 5 + native: false + projectedOutputColumns: [0] keys: fl_date (type: date) mode: hash outputColumnNames: _col0, _col1 @@ -1188,21 +1414,50 @@ STAGE PLANS: key expressions: _col0 (type: date) sort order: + Map-reduce partition columns: _col0 (type: date) + Reduce Sink Vectorization: + className: VectorReduceSinkMultiKeyOperator + native: true + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: bigint) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) + Group By Vectorization: + aggregators: VectorUDAFCountMerge(col 1) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 0 + native: false + projectedOutputColumns: [0] keys: KEY._col0 (type: date) mode: mergepartial outputColumnNames: _col0, _col1 Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -1400,17 +1655,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10- POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-30 07%3A00%3A00 POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-31 07%3A00%3A00 #### A masked pattern was here #### -Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00 -Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00 -Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00 -Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00 -Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00 -Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00 -Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00 -Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00 -Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00 -Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00 -Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00 +Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00 +Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00 +Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00 +Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00 +Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00 +Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00 +Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00 +Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00 +Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00 Baltimore New York 2010-10-21 17.0 1064 2010-10-21 07:00:00 Baltimore New York 2010-10-21 105.0 1142 2010-10-21 07:00:00 Baltimore New York 2010-10-21 28.0 1599 2010-10-21 07:00:00 @@ -1423,41 +1680,6 @@ Cleveland New York 2010-10-21 3.0 2630 2010-10-21 07:00:00 Cleveland New York 2010-10-21 29.0 2646 2010-10-21 07:00:00 Cleveland New York 2010-10-21 72.0 3014 2010-10-21 07:00:00 Washington New York 2010-10-21 22.0 7291 2010-10-21 07:00:00 -Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00 -Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00 -Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00 -Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00 -Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00 -Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00 -Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00 -Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00 -Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00 -Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00 -Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00 -Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00 -Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00 -Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00 -Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00 -Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00 -Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00 -Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00 -Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00 -Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00 -Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00 -Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00 -Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00 -Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00 -Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00 Baltimore New York 2010-10-25 -25.0 1064 2010-10-25 07:00:00 Baltimore New York 2010-10-25 92.0 1142 2010-10-25 07:00:00 Baltimore New York 2010-10-25 106.0 1599 2010-10-25 07:00:00 @@ -1470,19 +1692,42 @@ Cleveland New York 2010-10-25 -4.0 2630 2010-10-25 07:00:00 Cleveland New York 2010-10-25 81.0 2646 2010-10-25 07:00:00 Cleveland New York 2010-10-25 42.0 3014 2010-10-25 07:00:00 Washington New York 2010-10-25 9.0 7291 2010-10-25 07:00:00 -Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00 -Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00 -Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00 -Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00 -Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00 -Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00 -Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00 -Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00 -Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00 +Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00 +Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00 +Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00 +Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00 +Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00 +Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00 +Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00 +Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00 +Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00 +Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00 +Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00 +Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00 +Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00 +Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00 +Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00 +Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00 +Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00 +Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00 +Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00 +Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00 +Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00 +Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00 Baltimore New York 2010-10-27 -18.0 1064 2010-10-27 07:00:00 Baltimore New York 2010-10-27 49.0 1142 2010-10-27 07:00:00 Baltimore New York 2010-10-27 92.0 1599 2010-10-27 07:00:00 @@ -1506,18 +1751,6 @@ Cleveland New York 2010-10-28 3.0 2630 2010-10-28 07:00:00 Cleveland New York 2010-10-28 -6.0 2646 2010-10-28 07:00:00 Cleveland New York 2010-10-28 1.0 3014 2010-10-28 07:00:00 Washington New York 2010-10-28 45.0 7291 2010-10-28 07:00:00 -Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00 -Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00 -Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00 -Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00 -Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00 Baltimore New York 2010-10-30 14.0 272 2010-10-30 07:00:00 Baltimore New York 2010-10-30 -1.0 1805 2010-10-30 07:00:00 Baltimore New York 2010-10-30 5.0 3171 2010-10-30 07:00:00 @@ -1529,6 +1762,28 @@ Cleveland New York 2010-10-30 -23.0 2018 2010-10-30 07:00:00 Cleveland New York 2010-10-30 -12.0 2932 2010-10-30 07:00:00 Washington New York 2010-10-30 -27.0 5904 2010-10-30 07:00:00 Washington New York 2010-10-30 -16.0 5917 2010-10-30 07:00:00 +Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00 +Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00 +Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00 +Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00 +Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00 +Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00 +Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00 +Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00 +Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00 +Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00 +Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00 +Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00 +Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00 +Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00 +Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00 +Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00 +Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00 +Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00 +Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00 Baltimore New York 2010-10-31 -1.0 1599 2010-10-31 07:00:00 Baltimore New York 2010-10-31 -14.0 2571 2010-10-31 07:00:00 Chicago New York 2010-10-31 -25.0 361 2010-10-31 07:00:00 @@ -1638,26 +1893,67 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10- 2010-10-29 07:00:00 12 2010-10-30 07:00:00 11 2010-10-31 07:00:00 8 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_timestamp PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_timestamp POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: - Stage-0 is a root stage + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 STAGE PLANS: + Stage: Stage-1 + Tez +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: flights_tiny_orc_partitioned_timestamp + Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] + Select Operator + expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] + Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL + File Output Operator + compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false + Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Execution mode: vectorized, llap + LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true + Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: - TableScan - alias: flights_tiny_orc_partitioned_timestamp - Select Operator - expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp) - outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - ListSink + ListSink PREHOOK: query: select * from flights_tiny_orc_partitioned_timestamp PREHOOK: type: QUERY @@ -1691,17 +1987,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10- POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-30 07%3A00%3A00 POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-31 07%3A00%3A00 #### A masked pattern was here #### -Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00 -Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00 -Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00 -Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00 -Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00 -Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00 -Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00 -Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00 -Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00 -Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00 -Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00 +Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00 +Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00 +Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00 +Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00 +Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00 +Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00 +Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00 +Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00 +Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00 +Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00 Baltimore New York 2010-10-21 17.0 1064 2010-10-21 07:00:00 Baltimore New York 2010-10-21 105.0 1142 2010-10-21 07:00:00 Baltimore New York 2010-10-21 28.0 1599 2010-10-21 07:00:00 @@ -1714,41 +2012,6 @@ Cleveland New York 2010-10-21 3.0 2630 2010-10-21 07:00:00 Cleveland New York 2010-10-21 29.0 2646 2010-10-21 07:00:00 Cleveland New York 2010-10-21 72.0 3014 2010-10-21 07:00:00 Washington New York 2010-10-21 22.0 7291 2010-10-21 07:00:00 -Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00 -Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00 -Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00 -Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00 -Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00 -Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00 -Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00 -Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00 -Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00 -Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00 -Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00 -Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00 -Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00 -Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00 -Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00 -Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00 -Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00 -Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00 -Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00 -Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00 -Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00 -Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00 -Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00 -Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00 -Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00 -Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00 -Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00 -Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00 Baltimore New York 2010-10-25 -25.0 1064 2010-10-25 07:00:00 Baltimore New York 2010-10-25 92.0 1142 2010-10-25 07:00:00 Baltimore New York 2010-10-25 106.0 1599 2010-10-25 07:00:00 @@ -1761,19 +2024,42 @@ Cleveland New York 2010-10-25 -4.0 2630 2010-10-25 07:00:00 Cleveland New York 2010-10-25 81.0 2646 2010-10-25 07:00:00 Cleveland New York 2010-10-25 42.0 3014 2010-10-25 07:00:00 Washington New York 2010-10-25 9.0 7291 2010-10-25 07:00:00 -Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00 -Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00 -Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00 -Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00 -Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00 -Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00 -Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00 -Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00 -Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00 -Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00 +Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00 +Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00 +Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00 +Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00 +Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00 +Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00 +Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00 +Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00 +Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00 +Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00 +Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00 +Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00 +Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00 +Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00 +Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00 +Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00 +Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00 +Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00 +Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00 +Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00 +Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00 +Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00 +Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00 +Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00 +Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00 +Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00 Baltimore New York 2010-10-27 -18.0 1064 2010-10-27 07:00:00 Baltimore New York 2010-10-27 49.0 1142 2010-10-27 07:00:00 Baltimore New York 2010-10-27 92.0 1599 2010-10-27 07:00:00 @@ -1797,18 +2083,6 @@ Cleveland New York 2010-10-28 3.0 2630 2010-10-28 07:00:00 Cleveland New York 2010-10-28 -6.0 2646 2010-10-28 07:00:00 Cleveland New York 2010-10-28 1.0 3014 2010-10-28 07:00:00 Washington New York 2010-10-28 45.0 7291 2010-10-28 07:00:00 -Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00 -Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00 -Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00 -Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00 -Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00 -Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00 -Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00 Baltimore New York 2010-10-30 14.0 272 2010-10-30 07:00:00 Baltimore New York 2010-10-30 -1.0 1805 2010-10-30 07:00:00 Baltimore New York 2010-10-30 5.0 3171 2010-10-30 07:00:00 @@ -1820,6 +2094,28 @@ Cleveland New York 2010-10-30 -23.0 2018 2010-10-30 07:00:00 Cleveland New York 2010-10-30 -12.0 2932 2010-10-30 07:00:00 Washington New York 2010-10-30 -27.0 5904 2010-10-30 07:00:00 Washington New York 2010-10-30 -16.0 5917 2010-10-30 07:00:00 +Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00 +Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00 +Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00 +Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00 +Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00 +Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00 +Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00 +Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00 +Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00 +Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00 +Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00 +Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00 +Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00 +Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00 +Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00 +Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00 +Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00 +Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00 +Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00 +Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00 Baltimore New York 2010-10-31 -1.0 1599 2010-10-31 07:00:00 Baltimore New York 2010-10-31 -14.0 2571 2010-10-31 07:00:00 Chicago New York 2010-10-31 -25.0 361 2010-10-31 07:00:00 @@ -1828,12 +2124,16 @@ Chicago New York 2010-10-31 -4.0 1531 2010-10-31 07:00:00 Chicago New York 2010-10-31 -22.0 1610 2010-10-31 07:00:00 Chicago New York 2010-10-31 -15.0 3198 2010-10-31 07:00:00 Washington New York 2010-10-31 -18.0 7282 2010-10-31 07:00:00 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_timestamp sort by fl_num, fl_time limit 25 PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select * from flights_tiny_orc_partitioned_timestamp sort by fl_num, fl_time limit 25 POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -1852,46 +2152,102 @@ STAGE PLANS: TableScan alias: flights_tiny_orc_partitioned_timestamp Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL Reduce Output Operator key expressions: _col4 (type: int), _col5 (type: timestamp) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: date), _col3 (type: float) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: date), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: timestamp) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 4, 5, 0, 1] Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL Reduce Output Operator key expressions: _col4 (type: int), _col5 (type: timestamp) sort order: ++ + Reduce Sink Vectorization: + className: VectorReduceSinkOperator + native: false + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true + nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL TopN Hash Memory Usage: 0.1 value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: date), _col3 (type: float) Reducer 3 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: date), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: timestamp) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [2, 3, 4, 5, 0, 1] Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL Limit Number of rows: 25 + Limit Vectorization: + className: VectorLimitOperator + native: true Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL File Output Operator compressed: false + File Sink Vectorization: + className: VectorFileSinkOperator + native: false Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -1961,12 +2317,16 @@ Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00 Chicago New York 2010-10-25 -1.0 897 2010-10-25 07:00:00 Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00 Chicago New York 2010-10-27 -11.0 897 2010-10-27 07:00:00 -PREHOOK: query: explain +PREHOOK: query: explain vectorization expression select fl_time, count(*) from flights_tiny_orc_partitioned_timestamp group by fl_time PREHOOK: type: QUERY -POSTHOOK: query: explain +POSTHOOK: query: explain vectorization expression select fl_time, count(*) from flights_tiny_orc_partitioned_timestamp group by fl_time POSTHOOK: type: QUERY +PLAN VECTORIZATION: + enabled: true + enabledConditionsMet: [hive.vectorized.execution.enabled IS true] + STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 @@ -1984,12 +2344,26 @@ STAGE PLANS: TableScan alias: flights_tiny_orc_partitioned_timestamp Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE + TableScan Vectorization: + native: true + projectedOutputColumns: [0, 1, 2, 3, 4, 5] Select Operator expressions: fl_time (type: timestamp) outputColumnNames: fl_time + Select Vectorization: + className: VectorSelectOperator + native: true + projectedOutputColumns: [5] Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE Group By Operator aggregations: count() + Group By Vectorization: + aggregators: VectorUDAFCountStar(*) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 5 + native: false + projectedOutputColumns: [0] keys: fl_time (type: timestamp) mode: hash outputColumnNames: _col0, _col1 @@ -1998,21 +2372,50 @@ STAGE PLANS: key expressions: _col0 (type: timestamp) sort order: + Map-reduce partition columns: _col0 (type: timestamp) + Reduce Sink Vectorization: + className: VectorReduceSinkMultiKeyOperator + native: true + nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true Statistics: Num rows: 12 Data size: 576 Basic stats: COMPLETE Column stats: COMPLETE value expressions: _col1 (type: bigint) Execution mode: vectorized, llap LLAP IO: all inputs + Map Vectorization: + enabled: true + enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true + groupByVectorOutput: true + inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reducer 2 Execution mode: vectorized, llap + Reduce Vectorization: + enabled: true + enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true + groupByVectorOutput: true + allNative: false + usesVectorUDFAdaptor: false + vectorized: true Reduce Operator Tree: Group By Operator aggregations: count(VALUE._col0) + Group By Vectorization: + aggregators: VectorUDAFCountMerge(col 1) -> bigint + className: VectorGroupByOperator + vectorOutput: true + keyExpressions: col 0 + native: false + projectedOutputColumns: [0] keys: KEY._col0 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1 Statistics: Num rows: 12 Data size: 576 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false +