Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1797218BCB for ; Fri, 31 Jul 2015 22:19:05 +0000 (UTC) Received: (qmail 45658 invoked by uid 500); 31 Jul 2015 22:19:05 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 45633 invoked by uid 500); 31 Jul 2015 22:19:05 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 45623 invoked by uid 99); 31 Jul 2015 22:19:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2015 22:19:05 +0000 Date: Fri, 31 Jul 2015 22:19:04 +0000 (UTC) From: "Gopal V (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-11415) Add early termination for recursion in vectorization for deep filter queries MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-11415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649926#comment-14649926 ] Gopal V commented on HIVE-11415: -------------------------------- The right fix for this is to go ahead and take a ~8000 OR tree and turn it into a balanced tree ~14 levels deep. Failing to convert the tree to vectorization would be a bad idea in general, because this error can be progressively bypassed by running > Add early termination for recursion in vectorization for deep filter queries > ---------------------------------------------------------------------------- > > Key: HIVE-11415 > URL: https://issues.apache.org/jira/browse/HIVE-11415 > Project: Hive > Issue Type: Bug > Reporter: Prasanth Jayachandran > Assignee: Matt McCline > > Queries with deep filters (left deep) throws StackOverflowException in vectorization > {code} > Exception in thread "main" java.lang.StackOverflowError > at java.lang.Class.getAnnotation(Class.java:3415) > at org.apache.hive.common.util.AnnotationUtils.getAnnotation(AnnotationUtils.java:29) > at org.apache.hadoop.hive.ql.exec.vector.VectorExpressionDescriptor.getVectorExpressionClass(VectorExpressionDescriptor.java:332) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressionForUdf(VectorizationContext.java:988) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1164) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpression(VectorizationContext.java:439) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.createVectorExpression(VectorizationContext.java:1014) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getVectorExpressionForUdf(VectorizationContext.java:996) > at org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getGenericUdfVectorExpression(VectorizationContext.java:1164) > {code} > Sample query: > {code} > explain select count(*) from over1k where ( > (t=1 and si=2) > or (t=2 and si=3) > or (t=3 and si=4) > or (t=4 and si=5) > or (t=5 and si=6) > or (t=6 and si=7) > or (t=7 and si=8) > ... > .. > {code} > repeat the filter for few thousand times for reproduction of the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)