From dev-return-18823-apmail-openjpa-dev-archive=openjpa.apache.org@openjpa.apache.org Wed Mar 30 02:50:17 2011 Return-Path: Delivered-To: apmail-openjpa-dev-archive@www.apache.org Received: (qmail 80626 invoked from network); 30 Mar 2011 02:50:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 30 Mar 2011 02:50:16 -0000 Received: (qmail 49684 invoked by uid 500); 30 Mar 2011 02:50:16 -0000 Delivered-To: apmail-openjpa-dev-archive@openjpa.apache.org Received: (qmail 49655 invoked by uid 500); 30 Mar 2011 02:50:16 -0000 Mailing-List: contact dev-help@openjpa.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openjpa.apache.org Delivered-To: mailing list dev@openjpa.apache.org Received: (qmail 49644 invoked by uid 99); 30 Mar 2011 02:50:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 02:50:16 +0000 X-ASF-Spam-Status: No, hits=-1.0 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS,UNPARSEABLE_RELAY,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [148.87.113.121] (HELO rcsinet10.oracle.com) (148.87.113.121) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 02:50:10 +0000 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id p2U2nmEo013991 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Wed, 30 Mar 2011 02:49:49 GMT Received: from acsmt358.oracle.com (acsmt358.oracle.com [141.146.40.158]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id p2U2nlKQ008966 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 30 Mar 2011 02:49:47 GMT Received: from abhmt002.oracle.com (abhmt002.oracle.com [141.146.116.11]) by acsmt358.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id p2U2nkJU028866 for ; Tue, 29 Mar 2011 21:49:47 -0500 Received: from dhcp-rmdc-twvpn-2-vpnpool-10-159-63-144.vpn.oracle.com (/10.159.63.144) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 29 Mar 2011 19:49:46 -0700 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1084) Subject: Re: Stuck threads at Schema.getTable. From: Ravi Palacherla In-Reply-To: Date: Tue, 29 Mar 2011 20:49:45 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <4F05A725-8340-4ACF-9C89-EACC161DBB71@oracle.com> References: <1301281594502-6213602.post@n2.nabble.com> To: dev@openjpa.apache.org X-Mailer: Apple Mail (2.1084) X-Source-IP: acsmt358.oracle.com [141.146.40.158] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4D929A4B.011A,ss=1,fgs=0 The java source code might have been corrected in jdk6. I may not need to synchronize this in trunk but need to change 1.1.x = branch. I will spend more time and update this thread. Regards, Ravi. =20 On Mar 29, 2011, at 8:07 PM, Ravi Palacherla wrote: > Hi Mike, >=20 > As the java doc recommends that structural modification of = HashMap/TreeMap should be done only with proper synchronization. > The bug was not considered to be java bug but a solution to = synchronize the HashMap is recommended. >=20 > Hence I think we should synchronize the TreeMap in Schema.java to = avoid the infinite loop in TreeMap.get. >=20 > Here is a blog that explains the cause of this issue: > http://lightbody.net/blog/2005/07/hashmapget_can_cause_an_infini.html >=20 > Regards, > Ravi. >=20 > On Mar 28, 2011, at 12:52 PM, Michael Dick wrote: >=20 >> Hi Ravi, >>=20 >> It'd be nice to know which levels of Java it affects (I'm assuming = the bug >> is fixed in some level). If it's not planned for Java 6 then we = probably >> need to get the fix in. >>=20 >> -mike >>=20 >> On Mon, Mar 28, 2011 at 1:34 PM, Ravi Palacherla = >> wrote: >>=20 >>> Hi Mike, >>>=20 >>> The link used to work till yesterday. I will see if I can find = another link >>> that shows the issue. >>>=20 >>> It is not a sun bug but using an un-synchronized hashMap is = resulting in >>> the hashmap to get corrupted and it is spinning for ever at = hashmap.get >>> code. >>> 6423457 (coll) High cpu usage in HashMap.get() >>>=20 >>> This has been tracked as sun bug# 6423457 and the solution provided = is to >>> synchronize the hashMap. >>> Even though the issue might have been triggered because of = structural >>> modification of HashMap outside a synchronized block the thread dump = shows >>> that the thread is stuck at HashMap.get. >>>=20 >>> The same customer had a similar issue previously with = TableJDBCSeq.java: >>>=20 >>> java.lang.Thread.State: RUNNABLE >>> @ at java.util.HashMap.__tc_get(Unknown Source) >>> @ at java.util.HashMap.get(Unknown Source) >>> @ at >>> @ >>> = org.apache.openjpa.jdbc.kernel.TableJDBCSeq.getStatus(TableJDBCSeq.java:30= 8) >>> @ at >>> @ >>> = org.apache.openjpa.jdbc.kernel.TableJDBCSeq.nextInternal(TableJDBCSeq.java= :25 >>> @ 3) >>>=20 >>> This has been fixed by using concurrentHashMap as part of = OPENJPA-1765. >>>=20 >>>=20 >>> I think the issue is similar but with TreeMap in openJPA's = schema.java >>> code. >>>=20 >>> Here is a test code that I used to replicate the issue, but = unsuccessful so >>> far: >>>=20 >>> public void testSchemaTable() throws InterruptedException { >>> final Schema _schema =3D new SchemaGroup().addSchema("schema"); >>>=20 >>> for (int i=3D0;i>> _schema.addTable("Table:"+i); >>> count++; >>> } >>>=20 >>> Thread[] threads =3D new Thread[THREADS]; >>>=20 >>> for(int numThread=3D0; numThread< THREADS; numThread++) { >>> if( numThread % (THREADS-10) =3D=3D 0) { >>> threads[numThread] =3D new Thread(new Runnable() { >>> public void run() { >>> int lc =3D LOOPCOUNT+count; >>> for (int i=3Dcount;i>> _schema.addTable("Table:"+i); >>> } >>> } >>> }); >>> } else { >>> threads[numThread] =3D new Thread(new Runnable() { >>> public void run() { >>> int lc =3D LOOPCOUNT+count; >>> for (int i=3D0;i>> _schema.getTable("Table:"+i); >>> } >>> } >>> }); >>> } >>>=20 >>> } >>>=20 >>> for (int threadCount=3D0; threadCount>> threads[threadCount].start(); >>> } >>>=20 >>> for (int threadCount=3D0; threadCount>> threads[threadCount].join(); >>> } >>> } >>>=20 >>>=20 >>> Regards, >>> Ravi. >>>=20 >>> On Mar 28, 2011, at 11:00 AM, Michael Dick wrote: >>>=20 >>>> Hi Ravi, >>>>=20 >>>> The link didn't work for me, is this a bug in the TreeMap = implementation? >>>>=20 >>>> Multithreaded reads of a TreeMap should be fine. If a thread is = adding or >>>> deleting a node then we'll need to synchronize - is that what's = happening >>> in >>>> your case? >>>>=20 >>>> I've cross posted to dev - might get a better audience there. >>>>=20 >>>> -mike >>>>=20 >>>> On Sun, Mar 27, 2011 at 10:06 PM, Ravi P Palacherla < >>>> ravi.palacherla@oracle.com> wrote: >>>>=20 >>>>> Hi , >>>>>=20 >>>>> I see following stuck threads in openJPA code : >>>>>=20 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.util.TreeMap.getEntry(TreeMap.java:328) >>>>> at java.util.TreeMap.get(TreeMap.java:255) >>>>> at org.apache.openjpa.jdbc.schema.Schema.getTable(Schema.java:115) >>>>>=20 >>>>> I think the root cause of the issue is because of unsynchronized >>> TreeMap. >>>>>=20 >>>>> There is a sun issue that explains the same about a hashMap and I = think >>>>> this >>>>> is also because of the same issue explained in: >>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D6423457 >>>>>=20 >>>>> So, the fix is to synchronize the TreeMap but I do not have a = replicable >>>>> test case. >>>>>=20 >>>>> I created a simple testcase that will create around 100000 treeMap >>> entries >>>>> and then around 50 threads trying to access this treeMap but = unable to >>>>> replicate the issue. >>>>>=20 >>>>> Please suggest what you think should be done. >>>>> Can I just submit the code fix ( synchronizing the treemap) and = the test >>>>> case ( even though unable to replicate ) ? >>>>>=20 >>>>> Regards, >>>>> Ravi. >>>>>=20 >>>>> -- >>>>> View this message in context: >>>>>=20 >>> = http://openjpa.208410.n2.nabble.com/Stuck-threads-at-Schema-getTable-tp621= 3602p6213602.html >>>>> Sent from the OpenJPA Users mailing list archive at Nabble.com. >>>>>=20 >>>=20 >>>=20 >=20