From derby-dev-return-92424-apmail-db-derby-dev-archive=db.apache.org@db.apache.org Wed Dec 21 20:05:07 2011 Return-Path: X-Original-To: apmail-db-derby-dev-archive@www.apache.org Delivered-To: apmail-db-derby-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C603785C for ; Wed, 21 Dec 2011 20:05:07 +0000 (UTC) Received: (qmail 23292 invoked by uid 500); 21 Dec 2011 20:05:07 -0000 Delivered-To: apmail-db-derby-dev-archive@db.apache.org Received: (qmail 23273 invoked by uid 500); 21 Dec 2011 20:05:07 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 23266 invoked by uid 99); 21 Dec 2011 20:05:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Dec 2011 20:05:07 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [213.199.154.140] (HELO DB3EHSOBE002.bigfish.com) (213.199.154.140) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Dec 2011 20:04:57 +0000 Received: from mail59-db3-R.bigfish.com (10.3.81.242) by DB3EHSOBE002.bigfish.com (10.3.84.22) with Microsoft SMTP Server id 14.1.225.23; Wed, 21 Dec 2011 20:04:27 +0000 Received: from mail59-db3 (localhost [127.0.0.1]) by mail59-db3-R.bigfish.com (Postfix) with ESMTP id 86B3DC0383 for ; Wed, 21 Dec 2011 20:04:59 +0000 (UTC) X-SpamScore: -5 X-BigFish: VPS-5(zzbb2dI9371Ic85fh98dKzz1202hzz8275bh8275dhz2dh2a8h668h839h) X-Forefront-Antispam-Report: CIP:74.62.37.82;KIP:(null);UIP:(null);IPV:NLI;H:CPHUB1.canoga.com;RD:rrcs-74-62-37-82.west.biz.rr.com;EFVD:NLI Received: from mail59-db3 (localhost.localdomain [127.0.0.1]) by mail59-db3 (MessageSwitch) id 1324497899267473_13867; Wed, 21 Dec 2011 20:04:59 +0000 (UTC) Received: from DB3EHSMHS018.bigfish.com (unknown [10.3.81.252]) by mail59-db3.bigfish.com (Postfix) with ESMTP id 3DAFB660077 for ; Wed, 21 Dec 2011 20:04:59 +0000 (UTC) Received: from CPHUB1.canoga.com (74.62.37.82) by DB3EHSMHS018.bigfish.com (10.3.87.118) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 21 Dec 2011 20:04:25 +0000 Received: from CPHUB2.canoga.com (172.16.1.94) by CPHUB1.canoga.com (172.16.1.93) with Microsoft SMTP Server (TLS) id 8.2.213.0; Wed, 21 Dec 2011 12:05:51 -0800 Received: from vserver1.canoga.com ([169.254.2.87]) by CPHUB2.canoga.com ([172.16.1.94]) with mapi; Wed, 21 Dec 2011 12:05:51 -0800 From: "Bergquist, Brett" To: "derby-dev@db.apache.org" Date: Wed, 21 Dec 2011 12:04:29 -0800 Subject: RE: Problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1.1 Thread-Topic: Problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1.1 Thread-Index: AczAGWJBUpfxIgQyRX+Lidj1fIAxPAAAS9OA Message-ID: <97EB699F861AD841B5908C7CA9C9565601CC3A3958F3@VSERVER1.canoga.com> References: <97EB699F861AD841B5908C7CA9C9565601CC3A395890@VSERVER1.canoga.com> <4EF205B8.9040100@sbcglobal.net> <97EB699F861AD841B5908C7CA9C9565601CC3A3958A9@VSERVER1.canoga.com> <97EB699F861AD841B5908C7CA9C9565601CC3A3958EB@VSERVER1.canoga.com> <4EF2375C.6000305@sbcglobal.net> In-Reply-To: <4EF2375C.6000305@sbcglobal.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US X-TM-AS-Product-Ver: SMEX-8.0.0.1307-6.500.1024-18592.006 X-TM-AS-Result: No--21.440300-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No Content-Type: multipart/alternative; boundary="_000_97EB699F861AD841B5908C7CA9C9565601CC3A3958F3VSERVER1can_" MIME-Version: 1.0 X-OriginatorOrg: canoga.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_97EB699F861AD841B5908C7CA9C9565601CC3A3958F3VSERVER1can_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Nothing in the Derby log other than it logging a deadlock with the statemen= ts and a lock timeout with its statements and it indicating that cleanup ha= d started and completed. I will enable tracing with the documented (undocumented system property). = Thanks for that information! I will check for the XA transactions the next time I reproduce this. Maybe you could point me into the correct area to look. This seems to be t= riggered either through a lock timeout or a deadlock. The connection that= this is occurring through is an XA connection. I see the logging of this= in the server log but I am trying to find out where that would be logged f= rom. It seems after this occurs and because of the way connection pool is= being validated and recreated on error by Glassfish (configured to do so),= it gets into this state. What I don't understand is why this type of erro= r would cause the connection to appear to be invalid and I am trying to wor= k through both the Glassfish source and the Derby source to find out. The= connection is correctly handling other errors such as a duplication trying= to be inserted and this does not trigger the connection to appear to be in= valid. So I am trying to understand why a lock timeout or deadlock detec= tion might do so. This problem has only cropped up recently when they started performing mult= iple requests that I know have a deadlock path through them. I can fix tha= t problem later but this is a system level problem that I need to resolve. I really do appreciate the help and guidance and am willing to try to work = though this. I have to figure this out and either patch Glassfish or Derb= y in any case as my customer (think very very large wireless carrier) is ge= tting pretty PO'ed. From: Katherine Marsden [mailto:kmarsdenderby@sbcglobal.net] Sent: Wednesday, December 21, 2011 2:46 PM To: derby-dev@db.apache.org Subject: Re: Problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1= .1 On 12/21/2011 11:20 AM, Bergquist, Brett wrote: I'm having some trouble getting client side tracing to work. The connectio= ns are managed by Glassfish connection pool so I don't know where to set th= e traceDirectory and traceLevel properties. Can these be specified as pro= perties like the password, etc. They can be set on the connection URL or with undocumented system propert= ies, documented here #:) http://wiki.apache.org/db-derby/UndocumentedDerbyBehavior Looking at the info, again I am curious if there are corresponding server s= ide traces in the derby.log. Also it would be interesting to see if there are at this point any XA Trans= actions in need of recovery in the database. Just exit your application and connect with ij and run: select * from SYSCS_DIAG.TRANSACTION_TABLE ; --_000_97EB699F861AD841B5908C7CA9C9565601CC3A3958F3VSERVER1can_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

= Nothing in the Derby log other than it loggin= g a deadlock with the statements and a lock timeout with its statements and= it indicating that cleanup had started and completed.

 

I will enable traci= ng with the documented (undocumented system property).  Thanks for tha= t information!

 

I will check for the XA transactions the next time I reprod= uce this.

 

Maybe you could point me into the correct area to look.  Th= is seems to be triggered either through a lock timeout or a deadlock. =   The connection that this is occurring through is an XA connection.&n= bsp;  I see the logging of this in the server log but I am trying to f= ind out where that would be logged from.   It seems after this oc= curs and because of the way connection pool is being validated and recreate= d on error by Glassfish (configured to do so), it gets into this state.&nbs= p; What I don’t understand is why this type of error would cause the = connection to appear to be invalid and I am trying to work through both the= Glassfish source and the Derby source to find out.   The connect= ion is correctly handling other errors such as a duplication trying to be i= nserted and this does not trigger the connection to appear to be invalid.&n= bsp;   So I am trying to understand why a lock timeout or deadloc= k detection might do so.

 

This problem has only cropped up recently when th= ey started performing multiple requests that I know have a deadlock path th= rough them.  I can fix that problem later but this is a system level p= roblem that I need to resolve.

 

I really do appreciate the help and guidanc= e and am willing to try to work though this.   I have to figure t= his out and either patch Glassfish or Derby in any case as my customer (thi= nk very very large wireless carrier) is getting pretty PO’ed.

 = ;

From: Katherine Marsden [mailto:kmarsdenderby@sbcglobal.net]
= Sent: Wednesday, December 21, 2011 2:46 PM
To: derby-dev@d= b.apache.org
Subject: Re: Problem with a deadlock with Derby 10.8= .1.2 and Glassfish V2.1.1

 

On 12/21/2011 11:20 AM, Berg= quist, Brett wrote:

I’m having some trouble getting client side tracing to wor= k.  The connections are managed by Glassfish connection pool so I don&= #8217;t know where to set the traceDirectory and traceLevel properties.&nbs= p;  Can these be specified as properties like the password, etc.

They  = can be set on the connection URL or  with undocumented system properti= es, documented here #:)
http://wiki.apache.org/db-derby/UndocumentedDerbyBeha= vior

Looking at the info, again I am curious if there are corres= ponding server side traces in the derby.log.
Also it would be interestin= g to see if there are at this point any XA Transactions in need of recovery= in the database.
Just  exit your application and connect  wit= h ij and run:

select * from SYSCS_DIAG.TRANSACTION_TABLE ;

= --_000_97EB699F861AD841B5908C7CA9C9565601CC3A3958F3VSERVER1can_--