drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "arnab chatterjee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-3893) Issue with Drill after Hive Alters the Table
Date Mon, 05 Oct 2015 12:20:26 GMT
arnab chatterjee created DRILL-3893:
---------------------------------------

             Summary:  Issue with Drill after Hive Alters the Table
                 Key: DRILL-3893
                 URL: https://issues.apache.org/jira/browse/DRILL-3893
             Project: Apache Drill
          Issue Type: Bug
          Components: Functions - Hive, Storage - Hive
    Affects Versions: 1.1.0, 1.0.0
         Environment: DEV
            Reporter: arnab chatterjee


The application team reproduced this again on another partitioned table with existing data.

Providing some more details. I have enabled the version mode for errors. Drill is unable to
fetch the new column name that was introduced.This most likely to me seems  to me that it’s
still picking up the stale metadata of hive.


if (!tableColumns.contains(columnName)) {
            if (partitionNames.contains(columnName)) {
              selectedPartitionNames.add(columnName);
            } else {
              throw new ExecutionSetupException(String.format("Column %s does not exist",
columnName));
            }
          }


select testdata from allocation_test;
Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exist

Fragment 0:0

[Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on redlxd00006.nomura.com:31010]

  (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while initializing
HiveRecordReader: Column testdata does not exist
    org.apache.drill.exec.store.hive.HiveRecordReader.init():241
    org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
    org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745
  Caused By (org.apache.drill.common.exceptions.ExecutionSetupException) Column testdata does
not exist
    org.apache.drill.exec.store.hive.HiveRecordReader.init():206
    org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
    org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
    org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
    org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
    org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
    org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)

#########################################################

We have hit the issue again, this time the application team added a new column to the table
and once again Drill does not identify it.
Last time it was an update to an existing column name.

Please note that this is a partitioned table with existing data.

Does Drill Cache the Meta somewhere and hence it’s not getting reflected immediately ?

DRILL CLI
> select legaltickettext     from parentorders_asha;
Error: SYSTEM ERROR: ExecutionSetupException: Column legaltickettext          does not exist

Fragment 0:0

[Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on redlxd00006.nomura.com:31010] (state=,code=0)

HIVE CLI
hive> describe formatted parentorders_asha;
OK
# col_name              data_type               comment

recordtype              string
clientacronym           string
securityid              string
secdesc                 string
side                    string
orderqty                bigint
pricetype               string
price                   double
ordercapacity           string
exdestination           string
cashmargin              string
ordernotes              string
traderid                string
transacttime            timestamp
transacttimens          bigint
commonid                string
omssrc                  string
cumqty                  bigint
expirytype              string
orderid                 string
parentid                string
tradingaccount          string
businessarea            string
traderlocation          string
clientflowtype          string
complianceid            string
targetstrategy          string
expirydatetime          string
listid                  string
mergedto                string
legaltikcetref          string
legaltickettext         string

# Partition Information
# col_name              data_type               comment

tradedate               date
ordertradetype          string




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message