From derby-dev-return-99506-apmail-db-derby-dev-archive=db.apache.org@db.apache.org Mon Dec 10 22:41:24 2012 Return-Path: X-Original-To: apmail-db-derby-dev-archive@www.apache.org Delivered-To: apmail-db-derby-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C94EDEB2 for ; Mon, 10 Dec 2012 22:41:24 +0000 (UTC) Received: (qmail 48189 invoked by uid 500); 10 Dec 2012 22:41:22 -0000 Delivered-To: apmail-db-derby-dev-archive@db.apache.org Received: (qmail 48087 invoked by uid 500); 10 Dec 2012 22:41:22 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 47933 invoked by uid 99); 10 Dec 2012 22:41:22 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Dec 2012 22:41:22 +0000 Date: Mon, 10 Dec 2012 22:41:22 +0000 (UTC) From: "Brett Bergquist (JIRA)" To: derby-dev@db.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DERBY-5632) Logical deadlock happened when freezing/unfreezing the database MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DERBY-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528381#comment-13528381 ] Brett Bergquist commented on DERBY-5632: ---------------------------------------- This is still a major problem even when using the same connection, the database can get into a state that the freeze works, but the unfreeze locks up. At a customer size, this procedure is done each night to backup the database. Since the 9/1/2012, this has failed 5 times, causing the database to remain in a frozen state with no possibility of unfreezing it. When in this state, no connections to the database can be created including a connection to shut down the database. Forcefully killing the network server is the only way to recover. Note that there is a background job in the system that is running an UPDATE_STATISTICS every minute as in Derby 10.8.2.2, the automatic statistics update daemon has problems and cannot be used. > Logical deadlock happened when freezing/unfreezing the database > --------------------------------------------------------------- > > Key: DERBY-5632 > URL: https://issues.apache.org/jira/browse/DERBY-5632 > Project: Derby > Issue Type: Bug > Components: Documentation, Services > Affects Versions: 10.8.2.2 > Environment: Oracle M3000/Solaris 10 > Reporter: Brett Bergquist > Labels: derby_triage10_10 > Attachments: stack.txt > > > Tried to make a quick database backup by freezing the database, performing a ZFS snapshot, and then unfreezing the database. The database was frozen but then a connection to the database could not be established to unfreeze the database. > Looking at the stack trace of the network server, , I see 3 threads that are trying to process a connection request. Each of these is waiting on: > at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) > - waiting to lock <0xfffffffd3a7fcc68> (a org.apache.derby.impl.services.cache.ConcurrentCache) > That object is owned by: > - locked <0xfffffffd3a7fcc68> (a org.apache.derby.impl.services.cache.ConcurrentCache) > at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) > at org.apache.derby.impl.store.access.RAMTransaction.openGroupFetchScan(Unknown Source) > at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(Unknown Source) > at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.runExplicitly(Unknown Source) > at org.apache.derby.impl.sql.execute.AlterTableConstantAction.updateStatistics(Unknown Source) > which itself is waiting for the object: > at java.lang.Object.wait(Native Method) > - waiting on <0xfffffffd3ac1d608> (a org.apache.derby.impl.store.raw.log.LogToFile) > at java.lang.Object.wait(Object.java:485) > at org.apache.derby.impl.store.raw.log.LogToFile.flush(Unknown Source) > - locked <0xfffffffd3ac1d608> (a org.apache.derby.impl.store.raw.log.LogToFile) > at org.apache.derby.impl.store.raw.log.LogToFile.flush(Unknown Source) > at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.flush(Unknown Source) > So basically what I think is happening is that the database is frozen, the statistics are being updated on another thread which has the "org.apache.derby.impl.services.cache.ConcurrentCache" locked and then waits for the LogToFile lock and the connecting threads are waiting to lock "org.apache.derby.impl.services.cache.ConcurrentCache" to connect and these are where the database is going to be unfrozen. Not a deadlock as far as the JVM is concerned but it will never leave this state either. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira