Mailing-List: contact issues-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
Date: Thu, 28 Jan 2016 01:22:39 +0000 (UTC)
From: "Zameer Manji (JIRA)" <jira@apache.org>
To: issues@aurora.apache.org
Message-ID: <JIRA.12934139.1453823837000.231613.1453944159988@Atlassian.JIRA>
In-Reply-To: <JIRA.12934139.1453823837000@Atlassian.JIRA>
References: <JIRA.12934139.1453823837000@Atlassian.JIRA>
 <JIRA.12934139.1453823837920@arcas>
Subject: [jira] [Commented] (AURORA-1596) Invalid connection error from
 TaskHistoryPruner when using DBTaskStore
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/AURORA-1596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120570#comment-15120570 ] 

Zameer Manji commented on AURORA-1596:
--------------------------------------

Ignoring the lock acquisition then, what could be the cause of the underlying connection pool providing invalid connections causing this exception? Even after enabling all SQL/DB libraries to maximum verbosity, I have not been able to find anything in the logs that indicates how a connection is invalidated and why it is being used for a rollback.


> Invalid connection error from TaskHistoryPruner when using DBTaskStore
> ----------------------------------------------------------------------
>
>                 Key: AURORA-1596
>                 URL: https://issues.apache.org/jira/browse/AURORA-1596
>             Project: Aurora
>          Issue Type: Bug
>          Components: Scheduler
>            Reporter: Joshua Cohen
>            Assignee: Zameer Manji
>
> In a scheduler with the db task store enabled we're seeing this error causing frequent shutdowns:
> {noformat}
> Jan 26, 2016 3:30:00 PM com.google.common.util.concurrent.ServiceManager$ServiceListener failed
> SEVERE: Service TaskHistoryPruner [FAILED] has failed in the RUNNING state.
> org.apache.ibatis.exceptions.PersistenceException:
> ### Error rolling back transaction.  Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid.
> ### Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid.
>         at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
>         at org.apache.ibatis.session.defaults.DefaultSqlSession.rollback(DefaultSqlSession.java:216)
>         at org.apache.ibatis.session.SqlSessionManager.rollback(SqlSessionManager.java:299)
>         at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:116)
>         at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
>         at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:570)
>         at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:113)
>         at org.apache.aurora.scheduler.storage.Storage$Util.fetchTasks(Storage.java:297)
>         at org.apache.aurora.scheduler.pruning.TaskHistoryPruner.lambda$registerInactiveTask$103(TaskHistoryPruner.java:167)
>         at org.apache.aurora.scheduler.pruning.TaskHistoryPruner.lambda$failureNotifyingRunnable$101(TaskHistoryPruner.java:144)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid.
>         at org.apache.ibatis.datasource.pooled.PooledConnection.checkConnection(PooledConnection.java:254)
>         at org.apache.ibatis.datasource.pooled.PooledConnection.invoke(PooledConnection.java:243)
>         at com.sun.proxy.$Proxy131.getAutoCommit(Unknown Source)
>         at org.apache.ibatis.transaction.jdbc.JdbcTransaction.rollback(JdbcTransaction.java:79)
>         at org.apache.ibatis.executor.BaseExecutor.rollback(BaseExecutor.java:249)
>         at org.apache.ibatis.executor.CachingExecutor.rollback(CachingExecutor.java:119)
>         at org.apache.ibatis.session.defaults.DefaultSqlSession.rollback(DefaultSqlSession.java:213)
>         ... 15 more
> E0126 15:30:00.202 [AsyncProcessor-4, GuavaUtils$LifecycleShutdownListener:54] Service: TaskHistoryPruner [FAILED] failed unexpectedly. Triggering shutdown.
> I0126 15:30:00.202 [AsyncProcessor-4, Lifecycle:84] Shutting down application
> I0126 15:30:00.202 [AsyncProcessor-4, ShutdownRegistry$ShutdownRegistryImpl:77] Executing 2 shutdown commands.
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)