Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 70A88200C5B for ; Wed, 5 Apr 2017 00:34:50 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6F25C160BA1; Tue, 4 Apr 2017 22:34:50 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 42D2A160B90 for ; Wed, 5 Apr 2017 00:34:49 +0200 (CEST) Received: (qmail 73772 invoked by uid 500); 4 Apr 2017 22:34:48 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 73763 invoked by uid 99); 4 Apr 2017 22:34:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Apr 2017 22:34:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 1D3521A03BC for ; Tue, 4 Apr 2017 22:34:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id rHNqSagt-mVj for ; Tue, 4 Apr 2017 22:34:45 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id E685B5FE48 for ; Tue, 4 Apr 2017 22:34:43 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 189ABE09D6 for ; Tue, 4 Apr 2017 22:34:43 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3788224027 for ; Tue, 4 Apr 2017 22:34:42 +0000 (UTC) Date: Tue, 4 Apr 2017 22:34:42 +0000 (UTC) From: "Zelaine Fong (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (DRILL-5327) Hash aggregate can return empty batch which can cause schema change exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 04 Apr 2017 22:34:50 -0000 [ https://issues.apache.org/jira/browse/DRILL-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zelaine Fong reassigned DRILL-5327: ----------------------------------- Assignee: Jinfeng Ni (was: Boaz Ben-Zvi) > Hash aggregate can return empty batch which can cause schema change exception > ----------------------------------------------------------------------------- > > Key: DRILL-5327 > URL: https://issues.apache.org/jira/browse/DRILL-5327 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill > Affects Versions: 1.10.0 > Reporter: Chun Chang > Assignee: Jinfeng Ni > > Hash aggregate can return empty batches which cause drill to throw schema change exception (not handling this type of schema change). This is not a new bug. But a recent hash function change (a theoretically correct change) may have increased the chance of hitting this issue. I don't have scientific data to support my claim (in fact I don't believe it's the case), but a regular regression run used to pass fails now due to this bug. My concern is that existing drill users out there may have queries that used to work but fail now. It will be difficult to explain why the new release is better for them. I put this bug as blocker so we can discuss it before releasing 1.10. > {noformat} > /root/drillAutomation/framework-master/framework/resources/Advanced/tpcds/tpcds_sf1/original/text/query66.sql > Query: > -- start query 66 in stream 0 using template query66.tpl > SELECT w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > ship_carriers, > year1, > Sum(jan_sales) AS jan_sales, > Sum(feb_sales) AS feb_sales, > Sum(mar_sales) AS mar_sales, > Sum(apr_sales) AS apr_sales, > Sum(may_sales) AS may_sales, > Sum(jun_sales) AS jun_sales, > Sum(jul_sales) AS jul_sales, > Sum(aug_sales) AS aug_sales, > Sum(sep_sales) AS sep_sales, > Sum(oct_sales) AS oct_sales, > Sum(nov_sales) AS nov_sales, > Sum(dec_sales) AS dec_sales, > Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, > Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot, > Sum(mar_sales / w_warehouse_sq_ft) AS mar_sales_per_sq_foot, > Sum(apr_sales / w_warehouse_sq_ft) AS apr_sales_per_sq_foot, > Sum(may_sales / w_warehouse_sq_ft) AS may_sales_per_sq_foot, > Sum(jun_sales / w_warehouse_sq_ft) AS jun_sales_per_sq_foot, > Sum(jul_sales / w_warehouse_sq_ft) AS jul_sales_per_sq_foot, > Sum(aug_sales / w_warehouse_sq_ft) AS aug_sales_per_sq_foot, > Sum(sep_sales / w_warehouse_sq_ft) AS sep_sales_per_sq_foot, > Sum(oct_sales / w_warehouse_sq_ft) AS oct_sales_per_sq_foot, > Sum(nov_sales / w_warehouse_sq_ft) AS nov_sales_per_sq_foot, > Sum(dec_sales / w_warehouse_sq_ft) AS dec_sales_per_sq_foot, > Sum(jan_net) AS jan_net, > Sum(feb_net) AS feb_net, > Sum(mar_net) AS mar_net, > Sum(apr_net) AS apr_net, > Sum(may_net) AS may_net, > Sum(jun_net) AS jun_net, > Sum(jul_net) AS jul_net, > Sum(aug_net) AS aug_net, > Sum(sep_net) AS sep_net, > Sum(oct_net) AS oct_net, > Sum(nov_net) AS nov_net, > Sum(dec_net) AS dec_net > FROM (SELECT w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > 'ZOUROS' > || ',' > || 'ZHOU' AS ship_carriers, > d_year AS year1, > Sum(CASE > WHEN d_moy = 1 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS jan_sales, > Sum(CASE > WHEN d_moy = 2 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS feb_sales, > Sum(CASE > WHEN d_moy = 3 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS mar_sales, > Sum(CASE > WHEN d_moy = 4 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS apr_sales, > Sum(CASE > WHEN d_moy = 5 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS may_sales, > Sum(CASE > WHEN d_moy = 6 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS jun_sales, > Sum(CASE > WHEN d_moy = 7 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS jul_sales, > Sum(CASE > WHEN d_moy = 8 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS aug_sales, > Sum(CASE > WHEN d_moy = 9 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS sep_sales, > Sum(CASE > WHEN d_moy = 10 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS oct_sales, > Sum(CASE > WHEN d_moy = 11 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS nov_sales, > Sum(CASE > WHEN d_moy = 12 THEN ws_ext_sales_price * ws_quantity > ELSE 0 > END) AS dec_sales, > Sum(CASE > WHEN d_moy = 1 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS jan_net, > Sum(CASE > WHEN d_moy = 2 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS feb_net, > Sum(CASE > WHEN d_moy = 3 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS mar_net, > Sum(CASE > WHEN d_moy = 4 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS apr_net, > Sum(CASE > WHEN d_moy = 5 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS may_net, > Sum(CASE > WHEN d_moy = 6 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS jun_net, > Sum(CASE > WHEN d_moy = 7 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS jul_net, > Sum(CASE > WHEN d_moy = 8 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS aug_net, > Sum(CASE > WHEN d_moy = 9 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS sep_net, > Sum(CASE > WHEN d_moy = 10 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS oct_net, > Sum(CASE > WHEN d_moy = 11 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS nov_net, > Sum(CASE > WHEN d_moy = 12 THEN ws_net_paid_inc_ship * ws_quantity > ELSE 0 > END) AS dec_net > FROM web_sales, > warehouse, > date_dim, > time_dim, > ship_mode > WHERE ws_warehouse_sk = w_warehouse_sk > AND ws_sold_date_sk = d_date_sk > AND ws_sold_time_sk = t_time_sk > AND ws_ship_mode_sk = sm_ship_mode_sk > AND d_year = 1998 > AND t_time BETWEEN 7249 AND 7249 + 28800 > AND sm_carrier IN ( 'ZOUROS', 'ZHOU' ) > GROUP BY w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > d_year > UNION ALL > SELECT w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > 'ZOUROS' > || ',' > || 'ZHOU' AS ship_carriers, > d_year AS year1, > Sum(CASE > WHEN d_moy = 1 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS jan_sales, > Sum(CASE > WHEN d_moy = 2 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS feb_sales, > Sum(CASE > WHEN d_moy = 3 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS mar_sales, > Sum(CASE > WHEN d_moy = 4 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS apr_sales, > Sum(CASE > WHEN d_moy = 5 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS may_sales, > Sum(CASE > WHEN d_moy = 6 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS jun_sales, > Sum(CASE > WHEN d_moy = 7 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS jul_sales, > Sum(CASE > WHEN d_moy = 8 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS aug_sales, > Sum(CASE > WHEN d_moy = 9 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS sep_sales, > Sum(CASE > WHEN d_moy = 10 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS oct_sales, > Sum(CASE > WHEN d_moy = 11 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS nov_sales, > Sum(CASE > WHEN d_moy = 12 THEN cs_ext_sales_price * cs_quantity > ELSE 0 > END) AS dec_sales, > Sum(CASE > WHEN d_moy = 1 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS jan_net, > Sum(CASE > WHEN d_moy = 2 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS feb_net, > Sum(CASE > WHEN d_moy = 3 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS mar_net, > Sum(CASE > WHEN d_moy = 4 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS apr_net, > Sum(CASE > WHEN d_moy = 5 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS may_net, > Sum(CASE > WHEN d_moy = 6 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS jun_net, > Sum(CASE > WHEN d_moy = 7 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS jul_net, > Sum(CASE > WHEN d_moy = 8 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS aug_net, > Sum(CASE > WHEN d_moy = 9 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS sep_net, > Sum(CASE > WHEN d_moy = 10 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS oct_net, > Sum(CASE > WHEN d_moy = 11 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS nov_net, > Sum(CASE > WHEN d_moy = 12 THEN cs_net_paid * cs_quantity > ELSE 0 > END) AS dec_net > FROM catalog_sales, > warehouse, > date_dim, > time_dim, > ship_mode > WHERE cs_warehouse_sk = w_warehouse_sk > AND cs_sold_date_sk = d_date_sk > AND cs_sold_time_sk = t_time_sk > AND cs_ship_mode_sk = sm_ship_mode_sk > AND d_year = 1998 > AND t_time BETWEEN 7249 AND 7249 + 28800 > AND sm_carrier IN ( 'ZOUROS', 'ZHOU' ) > GROUP BY w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > d_year) x > GROUP BY w_warehouse_name, > w_warehouse_sq_ft, > w_city, > w_county, > w_state, > w_country, > ship_carriers, > year1 > ORDER BY w_warehouse_name > LIMIT 100 > Failed with exception > java.sql.SQLException: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema changes > Fragment 0:0 > [Error Id: 9af86933-491e-4ae4-b848-bf66cb4464f9 on atsqa6c88.qa.lab:31010] > at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489) > at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593) > at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215) > at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140) > at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:218) > at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:101) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema changes > Fragment 0:0 > [Error Id: 9af86933-491e-4ae4-b848-bf66cb4464f9 on atsqa6c88.qa.lab:31010] > at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123) > at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:343) > at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:88) > at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274) > at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244) > at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)