Return-Path: 
 <user-return-15073-apmail-hive-user-archive=hive.apache.org@hive.apache.org>
X-Original-To: apmail-hive-user-archive@www.apache.org
Delivered-To: apmail-hive-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFE1410611
	for <apmail-hive-user-archive@www.apache.org>;
 Wed, 18 Dec 2013 01:12:11 +0000 (UTC)
Received: (qmail 31505 invoked by uid 500); 18 Dec 2013 01:12:09 -0000
Delivered-To: apmail-hive-user-archive@hive.apache.org
Received: (qmail 31381 invoked by uid 500); 18 Dec 2013 01:12:09 -0000
Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hive.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hive.apache.org>
List-Post: <mailto:user@hive.apache.org>
List-Id: <user.hive.apache.org>
Reply-To: user@hive.apache.org
Delivered-To: mailing list user@hive.apache.org
Received: (qmail 31327 invoked by uid 99); 18 Dec 2013 01:12:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Dec 2013 01:12:09 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of spragues@gmail.com designates
 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com)
 (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Dec 2013 01:12:01 +0000
Received: by mail-qc0-f176.google.com with SMTP id i8so5530444qcq.21
        for <user@hive.apache.org>; Tue, 17 Dec 2013 17:11:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:from:date:message-id:subject:to:content-type;
        bh=d4uBZTn6hHgWCe2HrBKaVnLx5shQUJlR2OK7TXGtzYI=;
        b=LFyqx7fN4JYJYm8rOOI6NyL64PZeHfJQynDzVB73otjPq9MJ0tLo8wzR2l1H1/XC2X
         +jqWwhvbSVi4DJpZeSJ+9etQQOkwokYDceJhUvSRoY+tlTR278t5vLrg3SxARRzDJT/m
         u8LuYcoN5rPkDbiMh4eunbfcn9G1DR6cvNlNF0gdrU8js2/7esguViBYFgQT0bRDGqXU
         A5GE0T+DA6I6AMiRdHdGrDbskMf8nf8+CjLH8LmfasO9tyPK5+/j2ia9hVlKM39h+W+C
         b/5+lojE9oICJqu4zDCflCv1qKNM89dCbNc83iTfGgDL451i1VYBI2fuk99BK/Kzor/3
         to1w==
X-Received: by 10.49.132.65 with SMTP id os1mr48710246qeb.39.1387329100164;
 Tue, 17 Dec 2013 17:11:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.192.9 with HTTP; Tue, 17 Dec 2013 17:11:20 -0800 (PST)
From: Stephen Sprague <spragues@gmail.com>
Date: Tue, 17 Dec 2013 17:11:20 -0800
Message-ID: 
 <CAC06LGZQvdjL1dvb=2T72Pzzg9-oZWOr9Tpy2fAXcoq9ED0CgQ@mail.gmail.com>
Subject: metastore (mysql) connection problems - hive v0.12
To: "user@hive.apache.org" <user@hive.apache.org>
Content-Type: multipart/alternative; boundary=047d7bdca25833b44a04edc4ba3d
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7bdca25833b44a04edc4ba3d
Content-Type: text/plain; charset=ISO-8859-1

hi guys,
I'm pretty much at my wits end on this one. i'm using hive v0.12 against a
mysql metastore.   the error manifests itself (mostly randomly which is the
problem) as this error in the client:

2013-12-17 16:54:03,158 ERROR metastore.RetryingRawStore
(RetryingRawStore.java:invoke(146)) - JDO datastore error. Retrying
metastore command after 1000 ms (attempt 1 of 1)
2013-12-17 16:54:04,345 ERROR metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invoke(141)) -
MetaException(message:java.lang.RuntimeException: commitTransaction was
called but openTransactionCalls = 0. This probably indicates that there are
unbalanced calls to openTransaction/commitTransaction)


but i believe this to be just a symptom of something happening further
upstream.


examining the hive.log file for this useer (under /tmp/<user>/hive.log) i
find this hideous traceback with "bonecp" being the focus of attention.
There's nothing wrong with the mysql server metastore especially given it
works sometimes.  to me i think it has something to do with the connection
pooling logic but that's only speculation.

before i surrender and rollback to a previous version i ask:  Has anyone
encountered this before?  Or are there any nuggets in the traceback worthy
of insight?

thanks,
Stephen.


2013-12-17 16:54:03,142 ERROR bonecp.ConnectionHandle
(ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
Killing off all remaining connections in the connection pool. SQL State =
08S01
2013-12-17 16:54:03,144 ERROR bonecp.ConnectionHandle
(ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
Killing off all remaining connections in the connection pool. SQL State =
08007
2013-12-17 16:54:03,146 ERROR bonecp.ConnectionHandle
(ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
Killing off all remaining connections in the connection pool. SQL State =
08007
2013-12-17 16:54:03,147 ERROR bonecp.ConnectionHandle
(ConnectionHandle.java:markPossiblyBroken(297)) - Database access problem.
Killing off all remaining connections in the connection pool. SQL State =
08007
2013-12-17 16:54:03,148 ERROR bonecp.BoneCP
(BoneCP.java:terminateAllConnections(199)) - Error in attempting to close
connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.Util.getInstance(Util.java:386)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
        at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4730)
        at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4325)
        at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.getTransactionIsolation(ConnectionHandle.java:635)
        at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:422)
        at
org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:316)
        at
org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:300)
        at
org.datanucleus.store.rdbms.valuegenerator.SequenceTable.getNextVal(SequenceTable.java:194)
        at
org.datanucleus.store.rdbms.valuegenerator.TableGenerator.reserveBlock(TableGenerator.java:190)
        at
org.datanucleus.store.valuegenerator.AbstractGenerator.reserveBlock(AbstractGenerator.java:305)
        at
org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:98)
        at
org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerationBlock(AbstractGenerator.java:197)
        at
org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractGenerator.java:105)
        at
org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGenerator(RDBMSStoreManager.java:2019)
        at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1385)
        at
org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3727)
        at
org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2574)
        at
org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:526)
        at
org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:202)
        at
org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1326)
        at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2123)
        at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2212)
        at
org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:567)
        at
org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:326)
        at
org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:193)
        at
org.datanucleus.state.JDOStateManager.providedObjectField(JDOStateManager.java:1272)
        at
org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(MTable.java)
        at
org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(MTable.java)
        at
org.datanucleus.state.JDOStateManager.provideFields(JDOStateManager.java:1349)
        at
org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:289)
        at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
        at
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
        at
org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3777)
        at
org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3753)
        at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2124)
        at
org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1972)
        at
org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1820)
        at
org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
        at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
        at
org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
        at
org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:648)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
        at $Proxy4.createTable(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1074)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1107)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
        at $Proxy9.create_table_with_environment_context(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:482)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:471)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
        at $Proxy10.createTable(Unknown Source)
        at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:596)
        at
org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3661)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:252)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
        at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
        at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
        at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-12-17 16:54:03,153 ERROR bonecp.BoneCP
(BoneCP.java:terminateAllConnections(199)) - Error in attempting to close
connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.Util.getInstance(Util.java:386)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
        at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4730)
        at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4325)
        at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:404)
        at com.jolbox.bonecp.BoneCP.terminateAllConnections(BoneCP.java:197)
        at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:298)
        at
com.jolbox.bonecp.ConnectionHandle.getTransactionIsolation(ConnectionHandle.java:635)
        at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:422)
        at
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:356)
        at
org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:328)
        at
org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:64)
        at
org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:429)
        at
org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:395)
        at
org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:607)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
        at
org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
        at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:266)
        at
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:826)
        at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:760)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
        at $Proxy12.getTable(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1371)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
        at $Proxy13.get_table(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
        at $Proxy14.getTable(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:892)
        at
org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:245)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
        at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

--047d7bdca25833b44a04edc4ba3d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:courier =
new,monospace">hi guys,<br>I&#39;m pretty much at my wits end on this one. =
i&#39;m using hive v0.12 against a mysql metastore.=A0=A0 the error manifes=
ts itself (mostly randomly which is the problem) as this error in the clien=
t:<br>

<br>2013-12-17 16:54:03,158 ERROR metastore.RetryingRawStore (RetryingRawSt=
ore.java:invoke(146)) - JDO datastore error. Retrying metastore command aft=
er 1000 ms (attempt 1 of 1)<br>2013-12-17 16:54:04,345 ERROR metastore.Retr=
yingHMSHandler (RetryingHMSHandler.java:invoke(141)) - MetaException(messag=
e:java.lang.RuntimeException: commitTransaction was called but openTransact=
ionCalls =3D 0. This probably indicates that there are unbalanced calls to =
openTransaction/commitTransaction)<br>

<br><br></div><div class=3D"gmail_default" style=3D"font-family:courier new=
,monospace">but i believe this to be just a symptom of something happening =
further upstream.<br><br><br></div><div class=3D"gmail_default" style=3D"fo=
nt-family:courier new,monospace">

examining the hive.log file for this useer (under /tmp/&lt;user&gt;/hive.lo=
g) i find this hideous traceback with &quot;bonecp&quot; being the focus of=
 attention.=A0=A0 There&#39;s nothing wrong with the mysql server metastore=
 especially given it works sometimes.=A0 to me i think it has something to =
do with the connection pooling logic but that&#39;s only speculation.<br>

<br></div><div class=3D"gmail_default" style=3D"font-family:courier new,mon=
ospace">before i surrender and rollback to a previous version i ask:=A0 Has=
 anyone encountered this before?=A0 Or are there any nuggets in the traceba=
ck worthy of insight?<br>

<br></div><div class=3D"gmail_default" style=3D"font-family:courier new,mon=
ospace">thanks,<br></div><div class=3D"gmail_default" style=3D"font-family:=
courier new,monospace">Stephen.<br></div><div class=3D"gmail_default" style=
=3D"font-family:courier new,monospace">

<br></div><div class=3D"gmail_default" style=3D"font-family:courier new,mon=
ospace"><br>2013-12-17 16:54:03,142 ERROR bonecp.ConnectionHandle (Connecti=
onHandle.java:markPossiblyBroken(297)) - Database access problem. Killing o=
ff all remaining connections in the connection pool. SQL State =3D 08S01<br=
>

2013-12-17 16:54:03,144 ERROR bonecp.ConnectionHandle (ConnectionHandle.jav=
a:markPossiblyBroken(297)) - Database access problem. Killing off all remai=
ning connections in the connection pool. SQL State =3D 08007<br>2013-12-17 =
16:54:03,146 ERROR bonecp.ConnectionHandle (ConnectionHandle.java:markPossi=
blyBroken(297)) - Database access problem. Killing off all remaining connec=
tions in the connection pool. SQL State =3D 08007<br>

2013-12-17 16:54:03,147 ERROR bonecp.ConnectionHandle (ConnectionHandle.jav=
a:markPossiblyBroken(297)) - Database access problem. Killing off all remai=
ning connections in the connection pool. SQL State =3D 08007<br>2013-12-17 =
16:54:03,148 ERROR bonecp.BoneCP (BoneCP.java:terminateAllConnections(199))=
 - Error in attempting to close connection<br>

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Commu=
nications link failure during rollback(). Transaction resolution unknown.<b=
r>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeConstructorAccessorImpl.newIns=
tance0(Native Method)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeConstructorAccessorImpl.newInsta=
nce(NativeConstructorAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.=
reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor=
AccessorImpl.java:27)<br>

=A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Constructor.newInstance(Construc=
tor.java:513)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.Util.handleNewInst=
ance(Util.java:411)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.Util.getInst=
ance(Util.java:386)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.cre=
ateSQLException(SQLError.java:1013)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.createSQLException(SQLErro=
r.java:987)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.createSQLEx=
ception(SQLError.java:982)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLEr=
ror.createSQLException(SQLError.java:927)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionI=
mpl.java:4730)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.ConnectionImpl.re=
alClose(ConnectionImpl.java:4325)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdb=
c.ConnectionImpl.close(ConnectionImpl.java:1557)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:396)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.getTransactionI=
solation(ConnectionHandle.java:635)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanuc=
leus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(=
ConnectionFactoryImpl.java:422)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.SQLController.getState=
mentForQuery(SQLController.java:316)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanu=
cleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:300=
)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.valuegenerator.Se=
quenceTable.getNextVal(SequenceTable.java:194)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.valuegenerator.TableGe=
nerator.reserveBlock(TableGenerator.java:190)<br>=A0=A0=A0=A0=A0=A0=A0 at o=
rg.datanucleus.store.valuegenerator.AbstractGenerator.reserveBlock(Abstract=
Generator.java:305)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.valuegenerator.Abstrac=
tRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:98)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.valuegenerator.AbstractGener=
ator.obtainGenerationBlock(AbstractGenerator.java:197)<br>=A0=A0=A0=A0=A0=
=A0=A0 at org.datanucleus.store.valuegenerator.AbstractGenerator.next(Abstr=
actGenerator.java:105)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.RDBMSStoreManager.getS=
trategyValueForGenerator(RDBMSStoreManager.java:2019)<br>=A0=A0=A0=A0=A0=A0=
=A0 at org.datanucleus.store.AbstractStoreManager.getStrategyValue(Abstract=
StoreManager.java:1385)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.ExecutionContextImpl.newObjectId(E=
xecutionContextImpl.java:3727)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.=
state.JDOStateManager.setIdentity(JDOStateManager.java:2574)<br>=A0=A0=A0=
=A0=A0=A0=A0 at org.datanucleus.state.JDOStateManager.initialiseForPersiste=
ntNew(JDOStateManager.java:526)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.state.ObjectProviderFactoryImpl.ne=
wForPersistentNew(ObjectProviderFactoryImpl.java:202)<br>=A0=A0=A0=A0=A0=A0=
=A0 at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentN=
ew(ExecutionContextImpl.java:1326)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.ExecutionContextImpl.persistObject=
Internal(ExecutionContextImpl.java:2123)<br>=A0=A0=A0=A0=A0=A0=A0 at org.da=
tanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.j=
ava:2212)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.mapping.j=
ava.PersistableMapping.setObjectAsValue(PersistableMapping.java:567)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.mapping.java.Persistab=
leMapping.setObject(PersistableMapping.java:326)<br>=A0=A0=A0=A0=A0=A0=A0 a=
t org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField=
(ParameterSetter.java:193)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.state.JDOStateManager.providedObje=
ctField(JDOStateManager.java:1272)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.h=
adoop.hive.metastore.model.MTable.jdoProvideField(MTable.java)<br>=A0=A0=A0=
=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFie=
lds(MTable.java)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.state.JDOStateManager.provideField=
s(JDOStateManager.java:1349)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.st=
ore.rdbms.request.InsertRequest.execute(InsertRequest.java:289)<br>=A0=A0=
=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.inse=
rtTable(RDBMSPersistenceHandler.java:167)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.RDBMSPersistenceHandle=
r.insertObject(RDBMSPersistenceHandler.java:143)<br>=A0=A0=A0=A0=A0=A0=A0 a=
t org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateMana=
ger.java:3777)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.state.JDOStateMa=
nager.makePersistent(JDOStateManager.java:3753)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.ExecutionContextImpl.persistObject=
Internal(ExecutionContextImpl.java:2124)<br>=A0=A0=A0=A0=A0=A0=A0 at org.da=
tanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:=
1972)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.ExecutionContextImpl.pers=
istObject(ExecutionContextImpl.java:1820)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.ExecutionContextThreadedImpl.persi=
stObject(ExecutionContextThreadedImpl.java:217)<br>=A0=A0=A0=A0=A0=A0=A0 at=
 org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersist=
enceManager.java:727)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.api.jdo.JDOPersistenceManager.make=
Persistent(JDOPersistenceManager.java:752)<br>=A0=A0=A0=A0=A0=A0=A0 at org.=
apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:648)<=
br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Na=
tive Method)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(Native=
MethodAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.Delegat=
ingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br>=A0=
=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.RetryingRawStore.=
invoke(RetryingRawStore.java:124)<br>=A0=A0=A0=A0=A0=A0=A0 at $Proxy4.creat=
eTable(Unknown Source)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.m=
etastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1074=
)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMS=
Handler.create_table_with_environment_context(HiveMetaStore.java:1107)<br>=
=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ=
e Method)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.=
invoke(NativeMethodAccessorImpl.java:39)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(De=
legatingMethodAccessorImpl.java:25)<br>=A0=A0=A0=A0=A0=A0=A0 at java.lang.r=
eflect.Method.invoke(Method.java:597)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apach=
e.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:1=
03)<br>

=A0=A0=A0=A0=A0=A0=A0 at $Proxy9.create_table_with_environment_context(Unkn=
own Source)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.Hi=
veMetaStoreClient.createTable(HiveMetaStoreClient.java:482)<br>=A0=A0=A0=A0=
=A0=A0=A0 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTab=
le(HiveMetaStoreClient.java:471)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ=
e Method)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.=
invoke(NativeMethodAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.re=
flect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java=
:25)<br>

=A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)<b=
r>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.RetryingMetaSto=
reClient.invoke(RetryingMetaStoreClient.java:89)<br>=A0=A0=A0=A0=A0=A0=A0 a=
t $Proxy10.createTable(Unknown Source)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.metadata.Hive.createTabl=
e(Hive.java:596)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.exec=
.DDLTask.createTable(DDLTask.java:3661)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apa=
che.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:252)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Ta=
sk.java:151)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.Tas=
kRunner.runSequential(TaskRunner.java:65)<br>=A0=A0=A0=A0=A0=A0=A0 at org.a=
pache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.Driver.execute(Driver.ja=
va:1192)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.Driver.runIn=
ternal(Driver.java:1020)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive=
.ql.Driver.run(Driver.java:888)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hado=
op.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)<b=
r>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.service.ThriftHive$Processo=
r$execute.getResult(ThriftHive.java:644)<br>=A0=A0=A0=A0=A0=A0=A0 at org.ap=
ache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.=
java:628)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.ProcessFunction.pro=
cess(ProcessFunction.java:39)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.TBaseProcessor.process(TBaseProc=
essor.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.server.TThread=
PoolServer$WorkerProcess.run(TThreadPoolServer.java:206)<br>=A0=A0=A0=A0=A0=
=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool=
Executor.java:886)<br>

=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run=
(ThreadPoolExecutor.java:908)<br>=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.=
run(Thread.java:662)<br>2013-12-17 16:54:03,153 ERROR bonecp.BoneCP (BoneCP=
.java:terminateAllConnections(199)) - Error in attempting to close connecti=
on<br>

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Commu=
nications link failure during rollback(). Transaction resolution unknown.<b=
r>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeConstructorAccessorImpl.newIns=
tance0(Native Method)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeConstructorAccessorImpl.newInsta=
nce(NativeConstructorAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.=
reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructor=
AccessorImpl.java:27)<br>

=A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Constructor.newInstance(Construc=
tor.java:513)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.Util.handleNewInst=
ance(Util.java:411)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.Util.getInst=
ance(Util.java:386)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.cre=
ateSQLException(SQLError.java:1013)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.createSQLException(SQLErro=
r.java:987)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLError.createSQLEx=
ception(SQLError.java:982)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.SQLEr=
ror.createSQLException(SQLError.java:927)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionI=
mpl.java:4730)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdbc.ConnectionImpl.re=
alClose(ConnectionImpl.java:4325)<br>=A0=A0=A0=A0=A0=A0=A0 at com.mysql.jdb=
c.ConnectionImpl.close(ConnectionImpl.java:1557)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:396)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.internalClose(C=
onnectionHandle.java:404)<br>=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.Bon=
eCP.terminateAllConnections(BoneCP.java:197)<br>=A0=A0=A0=A0=A0=A0=A0 at co=
m.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:2=
98)<br>

=A0=A0=A0=A0=A0=A0=A0 at com.jolbox.bonecp.ConnectionHandle.getTransactionI=
solation(ConnectionHandle.java:635)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanuc=
leus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(=
ConnectionFactoryImpl.java:422)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.rdbms.ConnectionFactoryImpl$=
ManagedConnectionImpl.getXAResource(ConnectionFactoryImpl.java:356)<br>=A0=
=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.connection.ConnectionManagerImp=
l.allocateConnection(ConnectionManagerImpl.java:328)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.connection.AbstractConnectio=
nFactory.getConnection(AbstractConnectionFactory.java:64)<br>=A0=A0=A0=A0=
=A0=A0=A0 at org.datanucleus.store.AbstractStoreManager.getConnection(Abstr=
actStoreManager.java:429)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.AbstractStoreManager.getConn=
ection(AbstractStoreManager.java:395)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datan=
ucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:607)<br>=
=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.query.Query.executeQuery(Que=
ry.java:1786)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.store.query.Query.executeWithArray=
(Query.java:1672)<br>=A0=A0=A0=A0=A0=A0=A0 at org.datanucleus.api.jdo.JDOQu=
ery.execute(JDOQuery.java:266)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoo=
p.hive.metastore.ObjectStore.getMTable(ObjectStore.java:826)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.ObjectStore.getTa=
ble(ObjectStore.java:760)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMet=
hodAccessorImpl.invoke0(Native Method)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.refl=
ect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(De=
legatingMethodAccessorImpl.java:25)<br>=A0=A0=A0=A0=A0=A0=A0 at java.lang.r=
eflect.Method.invoke(Method.java:597)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apach=
e.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)<=
br>

=A0=A0=A0=A0=A0=A0=A0 at $Proxy12.getTable(Unknown Source)<br>=A0=A0=A0=A0=
=A0=A0=A0 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_=
table(HiveMetaStore.java:1371)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.Nati=
veMethodAccessorImpl.invoke0(Native Method)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(Native=
MethodAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.Delegat=
ingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br>=A0=
=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.RetryingHMSHandle=
r.invoke(RetryingHMSHandler.java:103)<br>=A0=A0=A0=A0=A0=A0=A0 at $Proxy13.=
get_table(Unknown Source)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hiv=
e.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ=
e Method)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.=
invoke(NativeMethodAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.re=
flect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java=
:25)<br>

=A0=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)<b=
r>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.metastore.RetryingMetaSto=
reClient.invoke(RetryingMetaStoreClient.java:89)<br>=A0=A0=A0=A0=A0=A0=A0 a=
t $Proxy14.getTable(Unknown Source)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(H=
ive.java:950)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.metadat=
a.Hive.getTable(Hive.java:892)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoo=
p.hive.ql.exec.MoveTask.execute(MoveTask.java:245)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Ta=
sk.java:151)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.exec.Tas=
kRunner.runSequential(TaskRunner.java:65)<br>=A0=A0=A0=A0=A0=A0=A0 at org.a=
pache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.Driver.execute(Driver.ja=
va:1192)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.ql.Driver.runIn=
ternal(Driver.java:1020)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive=
.ql.Driver.run(Driver.java:888)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hado=
op.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.cli.CliDriver.processCmd(Cl=
iDriver.java:216)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.cli.Cl=
iDriver.processLine(CliDriver.java:413)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apa=
che.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver=
.java:675)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hive.cli.CliDriver=
.main(CliDriver.java:614)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMet=
hodAccessorImpl.invoke0(Native Method)<br>

=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(Native=
MethodAccessorImpl.java:39)<br>=A0=A0=A0=A0=A0=A0=A0 at sun.reflect.Delegat=
ingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br>=A0=
=A0=A0=A0=A0=A0=A0 at java.lang.reflect.Method.invoke(Method.java:597)<br>

=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.util.RunJar.main(RunJar.java:208=
)<br><br></div></div>

--047d7bdca25833b44a04edc4ba3d--