Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EBCE9200B8B for ; Tue, 4 Oct 2016 08:31:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id EA6D2160AC9; Tue, 4 Oct 2016 06:31:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 156AC160AC5 for ; Tue, 4 Oct 2016 08:31:22 +0200 (CEST) Received: (qmail 85291 invoked by uid 500); 4 Oct 2016 06:31:22 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 85280 invoked by uid 99); 4 Oct 2016 06:31:22 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Oct 2016 06:31:22 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 1CFBB2C002D for ; Tue, 4 Oct 2016 06:31:22 +0000 (UTC) Date: Tue, 4 Oct 2016 06:31:22 +0000 (UTC) From: "Matt McCline (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-11394) Enhance EXPLAIN display for vectorization MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 04 Oct 2016 06:31:24 -0000 [ https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11394: -------------------------------- Description: Add detail to the EXPLAIN output showing why a Map or Reduce task was not vectorized. Add new VECTORIZATION option that displays 3 levels. Here are some examples: (At the beginning) {code} PLAN VECTORIZATION: enabled: true enabledConditionsMet: [hive.vectorized.execution.enabled IS true] {code} For Map and Reduce nodes: {code} Map Vectorization: enabled: true enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true groupByVectorOutput: false inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat allNative: false usesVectorUDFAdaptor: false vectorized: true {code} {code} Reduce Vectorization: enabled: true enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true notVectorizedReason: Aggregation Function UDF avg parameter expression for GROUPBY operator: Data type struct of Column[VALUE._col3] not supported vectorized: false {code} And, for each vectorized operator: {code} Select Vectorization: className: VectorSelectOperator native: true nativeConditionsMet: Supported IS true selectExpressions: IdentityExpression[6:decimal(38,18)] vectorized: true {code} {code} Map Join Vectorization: className: VectorMapJoinOperator native: false nativeConditionsMet: hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Supports Key Types IS true, When Fast Hash Table, then requires no Hybrid Hash Join IS true, Small table vectorizes IS true nativeConditionsNotMet: Not empty key IS false vectorized: true {code} The standard @Explain Annotation Type is used. A new 'vectorization' annotation marks each new class and method. Works for FORMATTED, like other non-vectorization variations. Consider adding options to just show Vectorization information: EXPLAIN VECTORIZATION [ONLY] [SUMMARY|DETAIL] where current patch is equivalent to EXPLAIN VECTORIZATION DETAIL. SUMMARY would just show PLAN VECTORIZATION and Map/Reduce Vectorization, but not operator detail. was: Add detail to the EXPLAIN output showing why a Map or Reduce task was not vectorized. Add new VECTORIZATION option that displays 3 levels. Here are some examples: (At the beginning) {code} PLAN VECTORIZATION: enabled: true enabledConditionsMet: [hive.vectorized.execution.enabled IS true] {code} For Map and Reduce nodes: {code} Map Vectorization: enabled: true enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true groupByVectorOutput: false inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat allNative: false usesVectorUDFAdaptor: false vectorized: true {code} {code} Reduce Vectorization: enabled: true enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true notVectorizedReason: Aggregation Function UDF avg parameter expression for GROUPBY operator: Data type struct of Column[VALUE._col3] not supported vectorized: false {code} And, for each vectorized operator: {code} Select Vectorization: className: VectorSelectOperator native: true nativeConditionsMet: Supported IS true selectExpressions: IdentityExpression[6:decimal(38,18)] vectorized: true {code} {code} Map Join Vectorization: className: VectorMapJoinOperator native: false nativeConditionsMet: hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Supports Key Types IS true, When Fast Hash Table, then requires no Hybrid Hash Join IS true, Small table vectorizes IS true nativeConditionsNotMet: Not empty key IS false vectorized: true {code} The standard @Explain Annotation Type is used. A new 'vectorization' annotation marks each new class and method. > Enhance EXPLAIN display for vectorization > ----------------------------------------- > > Key: HIVE-11394 > URL: https://issues.apache.org/jira/browse/HIVE-11394 > Project: Hive > Issue Type: Bug > Components: Hive > Reporter: Matt McCline > Assignee: Matt McCline > Priority: Critical > Attachments: HIVE-11394.01.patch > > > Add detail to the EXPLAIN output showing why a Map or Reduce task was not vectorized. > Add new VECTORIZATION option that displays 3 levels. Here are some examples: > (At the beginning) > {code} > PLAN VECTORIZATION: > enabled: true > enabledConditionsMet: [hive.vectorized.execution.enabled IS true] > {code} > For Map and Reduce nodes: > {code} > Map Vectorization: > enabled: true > enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true > groupByVectorOutput: false > inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat > allNative: false > usesVectorUDFAdaptor: false > vectorized: true > {code} > {code} > Reduce Vectorization: > enabled: true > enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true > notVectorizedReason: Aggregation Function UDF avg parameter expression for GROUPBY operator: Data type struct of Column[VALUE._col3] not supported > vectorized: false > {code} > And, for each vectorized operator: > {code} > Select Vectorization: > className: VectorSelectOperator > native: true > nativeConditionsMet: Supported IS true > selectExpressions: IdentityExpression[6:decimal(38,18)] > vectorized: true > {code} > {code} > Map Join Vectorization: > className: VectorMapJoinOperator > native: false > nativeConditionsMet: hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Supports Key Types IS true, When Fast Hash Table, then requires no Hybrid Hash Join IS true, Small table vectorizes IS true > nativeConditionsNotMet: Not empty key IS false > vectorized: true > {code} > The standard @Explain Annotation Type is used. A new 'vectorization' annotation marks each new class and method. > Works for FORMATTED, like other non-vectorization variations. > Consider adding options to just show Vectorization information: > EXPLAIN VECTORIZATION [ONLY] [SUMMARY|DETAIL] > where current patch is equivalent to EXPLAIN VECTORIZATION DETAIL. > SUMMARY would just show PLAN VECTORIZATION and Map/Reduce Vectorization, but not operator detail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)