From derby-dev-return-92427-apmail-db-derby-dev-archive=db.apache.org@db.apache.org Wed Dec 21 20:42:09 2011 Return-Path: X-Original-To: apmail-db-derby-dev-archive@www.apache.org Delivered-To: apmail-db-derby-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E5F07C90 for ; Wed, 21 Dec 2011 20:42:09 +0000 (UTC) Received: (qmail 88129 invoked by uid 500); 21 Dec 2011 20:42:09 -0000 Delivered-To: apmail-db-derby-dev-archive@db.apache.org Received: (qmail 88101 invoked by uid 500); 21 Dec 2011 20:42:09 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 88094 invoked by uid 99); 21 Dec 2011 20:42:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Dec 2011 20:42:09 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [66.94.237.89] (HELO nm24.access.bullet.mail.mud.yahoo.com) (66.94.237.89) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 21 Dec 2011 20:41:58 +0000 Received: from [66.94.237.201] by nm24.access.bullet.mail.mud.yahoo.com with NNFMP; 21 Dec 2011 20:41:31 -0000 Received: from [98.139.221.56] by tm12.access.bullet.mail.mud.yahoo.com with NNFMP; 21 Dec 2011 20:41:31 -0000 Received: from [127.0.0.1] by smtp109.sbc.mail.bf1.yahoo.com with NNFMP; 21 Dec 2011 20:41:30 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sbcglobal.net; s=s1024; t=1324500090; bh=KDJ4LHVK+SN7F6BUTZNlFCS2eXNsFOGGa3+H4ZgzYbo=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type; b=kPmHkTnh9x43vT2z0AkLsb2k2dUlmbPj6i/qtOs0VRq3pEcVT0z61MBL/KXDW1lC0JfySVJtJ1fuNAx0B6rkiIWK74Wk+NZIjzyJ6+wJud3NKfHsJCqsQrIbKsYUw4TS1hGPpVgvN+e7ArUCuow5/QhQ8FvqOsylLcuIBQH1o+o= X-Yahoo-Newman-Id: 630223.52488.bm@smtp109.sbc.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: zmevNXwVM1nrDGxAn32ooaCTGq4m9DZ4c2UptOtqsfosknd iNYDIgY7hHmvTvLp2hMVUADhWgkigfDJO4IWRh3E9GMyGt1R8h.J_QMLQ5N0 TJRC._OU99ddWYU7LhSbWkaJjL3Nirm23hPrBKEr.f343m3l6ddM7nKsfD98 OrxZWUzIKm1OkUQ7UF9Y6bVrUj2AvwuLVI9ueg2yrGlJYhNGky_2.7yqPiDv ZhoC6ajiUcp79ZSxIkZN452htjIgetocu7enHjACaM3vXAhLj5wuSGplj9.e mvmHW4nrae8dJ.mOGUgT6f.1dbMic5cGSjHzrboREpHwrx9p4CQAnbayXo0v ldwqXE07waS1vHCXy5TGx7P6aJp67kz7L5fBkxOd_pQ3KKZSMza1eostnY7N AdPf9ltujjL01XW2Sk9E1WvGd582YCETsrXtph.Ti1g-- X-Yahoo-SMTP: fBd8VESswBBwVkX.d9lIdXduzED_6kJxUAzIjM21tL._95FbORG0yg-- Received: from [192.168.0.15] (kmarsdenderby@99.159.45.105 with plain) by smtp109.sbc.mail.bf1.yahoo.com with SMTP; 21 Dec 2011 12:41:28 -0800 PST Message-ID: <4EF24477.5030607@sbcglobal.net> Date: Wed, 21 Dec 2011 12:41:27 -0800 From: Katherine Marsden User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: derby-dev@db.apache.org Subject: Re: Problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1.1 References: <97EB699F861AD841B5908C7CA9C9565601CC3A395890@VSERVER1.canoga.com> <4EF205B8.9040100@sbcglobal.net> <97EB699F861AD841B5908C7CA9C9565601CC3A3958A9@VSERVER1.canoga.com> <97EB699F861AD841B5908C7CA9C9565601CC3A3958EB@VSERVER1.canoga.com> <4EF2375C.6000305@sbcglobal.net> <97EB699F861AD841B5908C7CA9C9565601CC3A3958F3@VSERVER1.canoga.com> In-Reply-To: <97EB699F861AD841B5908C7CA9C9565601CC3A3958F3@VSERVER1.canoga.com> Content-Type: multipart/alternative; boundary="------------000203060508030904080608" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------000203060508030904080608 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 12/21/2011 12:04 PM, Bergquist, Brett wrote: > > Nothing in the Derby log other than it logging a deadlock with the > statements and a lock timeout with its statements and it indicating > that cleanup had started and completed. > > I will enable tracing with the documented (undocumented system > property). Thanks for that information! > > I will check for the XA transactions the next time I reproduce this. > > Maybe you could point me into the correct area to look. This seems to > be triggered either through a lock timeout or a deadlock. The > connection that this is occurring through is an XA connection. I see > the logging of this in the server log but I am trying to find out > where that would be logged from. It seems after this occurs and > because of the way connection pool is being validated and recreated on > error by Glassfish (configured to do so), it gets into this state. > What I don't understand is why this type of error would cause the > connection to appear to be invalid and I am trying to work through > both the Glassfish source and the Derby source to find out. The > connection is correctly handling other errors such as a duplication > trying to be inserted and this does not trigger the connection to > appear to be invalid. So I am trying to understand why a lock > timeout or deadlock detection might do so. > > This problem has only cropped up recently when they started performing > multiple requests that I know have a deadlock path through them. I > can fix that problem later but this is a system level problem that I > need to resolve. > > I really do appreciate the help and guidance and am willing to try to > work though this. I have to figure this out and either patch > Glassfish or Derby in any case as my customer (think very very large > wireless carrier) is getting pretty PO'ed. > The one thing I think of specifically with a deadlock is that it will automatically rollback the victim transaction and that might throw off this client logic regarding the state of the server. But I would think if there were just a simple problem with deadlocks it would have showed up before now. That said I don't see any specific tests in our XA tests: org.apache.derbyTesting.functionTests.tests.jdbapi.XATest or org.apache.derbyTesting.functionTests.tests.jdbcapi.XATransactionTest that test XAConnections with deadlocks. Is this a local transaction on an XA connection or a real XA transaction with two phase commit? You might want to try to test and an XAConnection with a simple deadlock case locally to see if that pops a reproduction. org.apache.derbyTesting.functionTests.tests.lang.DeadlockDetectionTest and org.apache.derbyTesting.functionTests.tests.lang have some examples of deadlocks. HTH, Kathey --------------000203060508030904080608 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 12/21/2011 12:04 PM, Bergquist, Brett wrote:

Nothing in the Derby log other than it logging a deadlock with the statements and a lock timeout with its statements and it indicating that cleanup had started and completed.

 

I will enable tracing with the documented (undocumented system property).  Thanks for that information!

 

I will check for the XA transactions the next time I reproduce this.

 

Maybe you could point me into the correct area to look.  This seems to be triggered either through a lock timeout or a deadlock.   The connection that this is occurring through is an XA connection.   I see the logging of this in the server log but I am trying to find out where that would be logged from.   It seems after this occurs and because of the way connection pool is being validated and recreated on error by Glassfish (configured to do so), it gets into this state.  What I don’t understand is why this type of error would cause the connection to appear to be invalid and I am trying to work through both the Glassfish source and the Derby source to find out.   The connection is correctly handling other errors such as a duplication trying to be inserted and this does not trigger the connection to appear to be invalid.    So I am trying to understand why a lock timeout or deadlock detection might do so.

 

This problem has only cropped up recently when they started performing multiple requests that I know have a deadlock path through them.  I can fix that problem later but this is a system level problem that I need to resolve.

 

I really do appreciate the help and guidance and am willing to try to work though this.   I have to figure this out and either patch Glassfish or Derby in any case as my customer (think very very large wireless carrier) is getting pretty PO’ed.

The one thing I think of specifically with a deadlock is that it will automatically rollback the victim transaction and that might throw off this client logic regarding the state of the server.    But I would think if there were just a simple problem with deadlocks it would have showed up before now. That said I don't see any specific tests in our XA tests: org.apache.derbyTesting.functionTests.tests.jdbapi.XATest or org.apache.derbyTesting.functionTests.tests.jdbcapi.XATransactionTest  that test XAConnections with deadlocks.

  Is this a local transaction on an XA connection or a real XA transaction with two phase commit? 

You might want to try to test and an XAConnection with  a simple deadlock case locally to see if that pops a reproduction.   org.apache.derbyTesting.functionTests.tests.lang.DeadlockDetectionTest and org.apache.derbyTesting.functionTests.tests.lang have  some examples of deadlocks.

HTH,

Kathey



--------------000203060508030904080608--