db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Olver <chris.ol...@3plearning.com>
Subject RE: Apache Derby - Locks up.
Date Fri, 06 Jun 2014 02:14:05 GMT
Hi Bertt,

I managed to replicate it by simply doing two queries after each other (both two simple select
* from). From lock up to traces starting to print was about 10 minutes. Stack traces below:

    (2014.6.6 11:41:35) Request fill DRDAConnThread_4 5

       RECEIVE BUFFER: EXCSAT              (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0076D04100010070  1041001B115E8485  .v.A...p.A...^..  ..}..........;de
0010   9982A88495839796  969360F960A38899  ..........`.`...  rbydncpool-9-thr
0020   85818460F1000911  6DC4859982A80022  ...`....m......"  ead-1..._Derby..
0030   115AC4D5C3F1F0F1  F0F061F1F04BF1F0  .Z........a..K..  .!DNC10100/10.10
0040   4BF24BF04060404D  F1F5F8F2F4F4F65D  K.K.@`@M.......]  .2.0 - (1582446)
0050   0018140414030007  24070007240F0007  ........$...$...  ................
0060   144000071C0804B8  000E1147D8C4C5D9  .@.........G....  . ..........QDER
0070   C2E861D1E5D40026  D00100020020106D  ..a....&..... .m  BY/JVM..}......_
0080   000611A200030016  211085A393404040  ........!....@@@  ...s......etl
0090   4040404040404040  40404040          @@@@@@@@@@@@


       (2014.6.6 11:41:50) Reply flush DRDAConnThread_4 5

       SEND BUFFER: EXCSATRD               (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0089D04200010083  1443001D115ED585  ...B.....C...^..  .i}....c.....;Ne
0010   A3A6969992E28599  A58599C39695A399  ................  tworkServerContr
0020   9693409481899500  1814041403000724  ..@............$  ol main.........
0030   070007240F000714  4000071C0804B800  ...$....@.......  ........ .......
0040   101147C197818388  8540C4859982A800  ..G......@......  ...Apache Derby.
0050   18116DD585A3A696  9992E28599A58599  ..m.............  .._NetworkServer
0060   C39695A399969300  22115AC3E2E2F1F0  ........".Z.....  Control...!CSS10
0070   F1F0F061F1F04BF1  F04BF24BF0406040  ...a..K..K.K.@`@  100/10.10.2.0 -
0080   4DF1F5F8F2F4F4F6  5D0010D002000200  M.......].......  (1582446)..}....
0090   0A14AC000611A200  03                .........         ......s..

       (2014.6.6 11:42:35) Request fill DRDAConnThread_3 5

       RECEIVE BUFFER: EXCSAT              (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0077D04100010071  1041001C115E8485  .w.A...q.A...^..  ..}..........;de
0010   9982A88495839796  969360F1F060A388  ..........`..`..  rbydncpool-10-th
0020   9985818460F10009  116DC4859982A800  ....`....m......  read-1..._Derby.
0030   22115AC4D5C3F1F0  F1F0F061F1F04BF1  ".Z........a..K.  ..!DNC10100/10.1
0040   F04BF24BF0406040  4DF1F5F8F2F4F4F6  .K.K.@`@M.......  0.2.0 - (1582446
0050   5D00181404140300  0724070007240F00  ]........$...$..  )...............
0060   07144000071C0804  B8000E1147D8C4C5  ..@.........G...  .. ..........QDE
0070   D9C2E861D1E5D400  26D0010002002010  ...a....&..... .  RBY/JVM..}......
0080   6D000611A2000300  16211085A3934040  m........!....@@  _...s......etl
0090   4040404040404040  4040404040        @@@@@@@@@@@@@


       (2014.6.6 11:42:59) Reply flush DRDAConnThread_3 5

       SEND BUFFER: EXCSATRD               (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0089D04200010083  1443001D115ED585  ...B.....C...^..  .i}....c.....;Ne
0010   A3A6969992E28599  A58599C39695A399  ................  tworkServerContr
0020   9693409481899500  1814041403000724  ..@............$  ol main.........
0030   070007240F000714  4000071C0804B800  ...$....@.......  ........ .......
0040   101147C197818388  8540C4859982A800  ..G......@......  ...Apache Derby.
0050   18116DD585A3A696  9992E28599A58599  ..m.............  .._NetworkServer
0060   C39695A399969300  22115AC3E2E2F1F0  ........".Z.....  Control...!CSS10
0070   F1F0F061F1F04BF1  F04BF24BF0406040  ...a..K..K.K.@`@  100/10.10.2.0 -
0080   4DF1F5F8F2F4F4F6  5D0010D002000200  M.......].......  (1582446)..}....
0090   0A14AC000611A200  03                .........         ......s..

       (2014.6.6 11:43:46) Request fill DRDAConnThread_4 5

       RECEIVE BUFFER: EXCSAT              (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0077D04100010071  1041001C115E8485  .w.A...q.A...^..  ..}..........;de
0010   9982A88495839796  969360F1F160A388  ..........`..`..  rbydncpool-11-th
0020   9985818460F10009  116DC4859982A800  ....`....m......  read-1..._Derby.
0030   22115AC4D5C3F1F0  F1F0F061F1F04BF1  ".Z........a..K.  ..!DNC10100/10.1
0040   F04BF24BF0406040  4DF1F5F8F2F4F4F6  .K.K.@`@M.......  0.2.0 - (1582446
0050   5D00181404140300  0724070007240F00  ]........$...$..  )...............
0060   07144000071C0804  B8000E1147D8C4C5  ..@.........G...  .. ..........QDE
0070   D9C2E861D1E5D400  26D0010002002010  ...a....&..... .  RBY/JVM..}......
0080   6D000611A2000300  16211085A3934040  m........!....@@  _...s......etl
0090   4040404040404040  4040404040        @@@@@@@@@@@@@


       (2014.6.6 11:43:57) Reply flush DRDAConnThread_4 5

       SEND BUFFER: EXCSATRD               (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0089D04200010083  1443001D115ED585  ...B.....C...^..  .i}....c.....;Ne
0010   A3A6969992E28599  A58599C39695A399  ................  tworkServerContr
0020   9693409481899500  1814041403000724  ..@............$  ol main.........
0030   070007240F000714  4000071C0804B800  ...$....@.......  ........ .......
0040   101147C197818388  8540C4859982A800  ..G......@......  ...Apache Derby.
0050   18116DD585A3A696  9992E28599A58599  ..m.............  .._NetworkServer
0060   C39695A399969300  22115AC3E2E2F1F0  ........".Z.....  Control...!CSS10
0070   F1F0F061F1F04BF1  F04BF24BF0406040  ...a..K..K.K.@`@  100/10.10.2.0 -
0080   4DF1F5F8F2F4F4F6  5D0010D002000200  M.......].......  (1582446)..}....
0090   0A14AC000611A200  03                .........         ......s..


       (2014.6.6 11:44:9) Request fill DRDAConnThread_4 5

       (2014.6.6 11:45:53) Request fill DRDAConnThread_3 5

       RECEIVE BUFFER: EXCSAT              (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0077D04100010071  1041001C115E8485  .w.A...q.A...^..  ..}..........;de
0010   9982A88495839796  969360F1F260A388  ..........`..`..  rbydncpool-12-th
0020   9985818460F10009  116DC4859982A800  ....`....m......  read-1..._Derby.
0030   22115AC4D5C3F1F0  F1F0F061F1F04BF1  ".Z........a..K.  ..!DNC10100/10.1
0040   F04BF24BF0406040  4DF1F5F8F2F4F4F6  .K.K.@`@M.......  0.2.0 - (1582446
0050   5D00181404140300  0724070007240F00  ]........$...$..  )...............
0060   07144000071C0804  B8000E1147D8C4C5  ..@.........G...  .. ..........QDE
0070   D9C2E861D1E5D400  26D0010002002010  ...a....&..... .  RBY/JVM..}......
0080   6D000611A2000300  16211085A3934040  m........!....@@  _...s......etl
0090   4040404040404040  4040404040        @@@@@@@@@@@@@


       (2014.6.6 11:46:2) Reply flush DRDAConnThread_3 5

       SEND BUFFER: EXCSATRD               (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   0089D04200010083  1443001D115ED585  ...B.....C...^..  .i}....c.....;Ne
0010   A3A6969992E28599  A58599C39695A399  ................  tworkServerContr
0020   9693409481899500  1814041403000724  ..@............$  ol main.........
0030   070007240F000714  4000071C0804B800  ...$....@.......  ........ .......
0040   101147C197818388  8540C4859982A800  ..G......@......  ...Apache Derby.
0050   18116DD585A3A696  9992E28599A58599  ..m.............  .._NetworkServer
0060   C39695A399969300  22115AC3E2E2F1F0  ........".Z.....  Control...!CSS10
0070   F1F0F061F1F04BF1  F04BF24BF0406040  ...a..K..K.K.@`@  100/10.10.2.0 -
0080   4DF1F5F8F2F4F4F6  5D0010D002000200  M.......].......  (1582446)..}....
0090   0A14AC000611A200  03                .........         ......s..


       (2014.6.6 11:46:11) Request fill DRDAConnThread_3 5

       (2014.6.6 12:0:36) Request fill DRDAConnThread_4 5

       RECEIVE BUFFER:                     (ASCII)           (EBCDIC)
       0 1 2 3 4 5 6 7   8 9 A B C D E F   0123456789ABCDEF  0123456789ABCDEF
0000   434D443A00010000  06                CMD:.....         .(.......


Within the Console, shortly after the trace I got this:

Fri Jun 06 12:06:57 EST 2014 Thread[DRDAConnThread_2,5,main] (XID = 239547), (SESSIONID =
19), (DATABASE = etl), (DRDAID = ????????.????-449232660812100998{12}), Cleanup action starting
Fri Jun 06 12:08:18 EST 2014 Thread[DRDAConnThread_2,5,main] (XID = 239547), (SESSIONID =
19), (DATABASE = etl), (DRDAID = ????????.????-449232660812100998{12}), Failed Statement is:
SELECT * FROM APP.PROFILES
java.lang.OutOfMemoryError: Java heap space
                at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
                at java.util.HashMap.putVal(HashMap.java:630)
                at java.util.HashMap.put(HashMap.java:611)
                at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:146)
                at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:128)
                at sun.util.resources.OpenListResourceBundle.handleKeySet(OpenListResourceBundle.java:96)
                at java.util.ResourceBundle.containsKey(ResourceBundle.java:1807)
                at sun.util.locale.provider.LocaleResources.getTimeZoneNames(LocaleResources.java:262)
                at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayNameArray(TimeZoneNameProviderImpl.java:122)
                at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayName(TimeZoneNameProviderImpl.java:98)
                at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getName(TimeZoneNameUtility.java:325)
                at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:281)
                at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:267)
                at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
                at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
                at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(TimeZoneNameUtility.java:135)
                at java.util.TimeZone.getDisplayName(TimeZone.java:400)
                at java.util.Date.toString(Date.java:1045)
                at java.lang.String.valueOf(String.java:2979)
                at java.lang.StringBuffer.append(StringBuffer.java:265)
                at org.apache.derby.impl.services.stream.BasicGetLogHeader.getHeader(Unknown
Source)
                at org.apache.derby.iapi.error.ErrorStringBuilder.appendln(Unknown Source)
                at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown
Source)
                at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(Unknown
Source)
                at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
                at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
                at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
                at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown
Source)
                at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
                at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.writeFDODTA(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.writeQRYDTA(Unknown Source)


Any help is appreciated.

Chris

From: Bergquist, Brett [mailto:BBergquist@canoga.com]
Sent: 06 June 2014 03:52
To: Derby Discussion
Subject: RE: Apache Derby - Locks up.

Chris, can you get a stack trace from the Network Server process when this occurs?

From: Chris Olver [mailto:chris.olver@3plearning.com]
Sent: Wednesday, June 04, 2014 4:42 AM
To: derby-user@db.apache.org<mailto:derby-user@db.apache.org>
Subject: Apache Derby - Locks up.

Hi,

We are looking to use Apache Derby (Network Server) as a Caching Layer for our ETL process.
Unfortunately we are finding that it locks up quite frequently. We have two tables which have
around 10 million rows, two indexes in each. We can be reading (straight SELECT * FROM) or
writing updates when it happens.

CPU will spike to 100% (its on a rather powerful box) and then all existing and new JDBC clients
are unable to connect. Running runtimeinfo (when it locks up, issuing this command can take
a few minutes to get a response):

--- Derby Network Server Runtime Information ---
---------- Session Information ---------------
Session # :116
Database :etl
User : abc
# Statements:1
Prepared Statement Information:
        Stmt ID         SQLText
        -------------   -----------
        SYSLH0001       SELECT * FROM APP.USERS

Session # :117
-------------------------------------------------------------
# Connection Threads : 4
# Active Sessions : 2
# Waiting  Sessions : 0

Total Memory : 1756889088       Free Memory : 306272128

No errors can be seen in the log. I am rather confused as it seems like the perfect solution
Derby just locks up.

Thoughts or advise appreciated.

OS: Windows 8.1
Java Runtime: 1.8.0_05-b13
Derby: 10.10.2.0

Regards,

-Chris

Mime
View raw message