Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EE2CF18918 for ; Mon, 5 Oct 2015 12:21:00 +0000 (UTC) Received: (qmail 87605 invoked by uid 500); 5 Oct 2015 12:20:26 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 87568 invoked by uid 500); 5 Oct 2015 12:20:26 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 87558 invoked by uid 99); 5 Oct 2015 12:20:26 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2015 12:20:26 +0000 Date: Mon, 5 Oct 2015 12:20:26 +0000 (UTC) From: "arnab chatterjee (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (DRILL-3893) Issue with Drill after Hive Alters the Table MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 arnab chatterjee created DRILL-3893: --------------------------------------- Summary: Issue with Drill after Hive Alters the Table Key: DRILL-3893 URL: https://issues.apache.org/jira/browse/DRILL-3893 Project: Apache Drill Issue Type: Bug Components: Functions - Hive, Storage - Hive Affects Versions: 1.1.0, 1.0.0 Environment: DEV Reporter: arnab chatterjee The application team reproduced this again on another partitioned table wit= h existing data. Providing some more details. I have enabled the version mode for errors. Dr= ill is unable to fetch the new column name that was introduced.This most li= kely to me seems to me that it=E2=80=99s still picking up the stale metada= ta of hive. if (!tableColumns.contains(columnName)) { if (partitionNames.contains(columnName)) { selectedPartitionNames.add(columnName); } else { throw new ExecutionSetupException(String.format("Column %s do= es not exist", columnName)); } } select testdata from allocation_test; Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exis= t Fragment 0:0 [Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on redlxd00006.nomura.com:3= 1010] (org.apache.drill.common.exceptions.ExecutionSetupException) Failure whil= e initializing HiveRecordReader: Column testdata does not exist org.apache.drill.exec.store.hive.HiveRecordReader.init():241 org.apache.drill.exec.store.hive.HiveRecordReader.():138 org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58 org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34 org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150 org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173 org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106 org.apache.drill.exec.physical.impl.ImplCreator.getExec():81 org.apache.drill.exec.work.fragment.FragmentExecutor.run():235 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1142 java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Co= lumn testdata does not exist org.apache.drill.exec.store.hive.HiveRecordReader.init():206 org.apache.drill.exec.store.hive.HiveRecordReader.():138 org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58 org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34 org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150 org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173 org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106 org.apache.drill.exec.physical.impl.ImplCreator.getExec():81 org.apache.drill.exec.work.fragment.FragmentExecutor.run():235 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1142 java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 (state=3D,code=3D0) ######################################################### We have hit the issue again, this time the application team added a new col= umn to the table and once again Drill does not identify it. Last time it was an update to an existing column name. Please note that this is a partitioned table with existing data. Does Drill Cache the Meta somewhere and hence it=E2=80=99s not getting refl= ected immediately ? DRILL CLI > select legaltickettext from parentorders_asha; Error: SYSTEM ERROR: ExecutionSetupException: Column legaltickettext = does not exist Fragment 0:0 [Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on redlxd00006.nomura.com:3= 1010] (state=3D,code=3D0) HIVE CLI hive> describe formatted parentorders_asha; OK # col_name data_type comment recordtype string clientacronym string securityid string secdesc string side string orderqty bigint pricetype string price double ordercapacity string exdestination string cashmargin string ordernotes string traderid string transacttime timestamp transacttimens bigint commonid string omssrc string cumqty bigint expirytype string orderid string parentid string tradingaccount string businessarea string traderlocation string clientflowtype string complianceid string targetstrategy string expirydatetime string listid string mergedto string legaltikcetref string legaltickettext string # Partition Information # col_name data_type comment tradedate date ordertradetype string -- This message was sent by Atlassian JIRA (v6.3.4#6332)