jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Hobi <d.h...@gmx.ch>
Subject Re: JVM hangs during startup (indexing) (2)
Date Fri, 08 Mar 2013 10:09:00 GMT
> Does it happen also on another machine? 

Yes, but only once (of 5…6 reindexing is running)

> Try latest Oracle JDK to see if it makes any difference
Same effect with latest Oracle JVM (JRE)

CPU / IO looks ok to me as long as the JVM is running.
As soon as the JVM freezes/hangs CPU/IO is idle

root@miraculix:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15774
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 9000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15774
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Thanks for your input so far!
Daniel

Am 08.03.2013 um 10:49 schrieb <mslama@email.cz>:

> It is really weird that JVM process is frozen and you cannot get thread 
> dump. I never saw such problem. It looks like HW or OS problem.
> Does it happen also on another machine? From your log I also see you use 
> OpenJDK. Try latest Oracle JDK to see if it makes any difference.
> 
> 
> 
> 
> If you monitor frozen JVM process is there any activity? (CPU/io). Is there 
> any OS limit on process mem size? Check ulimit settings. (ulimit -a)
> 
> 
> 
> 
> Marek
> 
> 
> "Hi Marek
> 
> I use jstack -F <pid> to get thread dumps.
> - Except once the output of jstack is the only one I get (= the first dump I
> provided) 
> - Once (the jvm was not completely frozen, just the indexing made no further
> progress) jstack also made a dump in our log file (= second dump)
> 
> Another result after killing the jvm process with "kill -4 <pid>" (kill -3 
> does not work due to a frozen jvm)
> http://pastebin.com/qc2bUkbk
> 
> Daniel
> 
> Am 08.03.2013 um 10:16 schrieb <mslama@email.cz>:
> 
>> I checked 2 thread dumps you provided so I can only comment what I saw 
>> there. I did not see any deadlock. Only
>> problem I saw was in main thread where it was waiting for connection.
>> 
>> 
>> 
>> 
>> 
>> 1. 
>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
>> 00007fefd3812000]
>> 2. 
>> java.lang.Thread.State: WAITING (on object monitor)
>> 3. 
>> at java.lang.Object.$$YJP$$wait(Native Method)
>> 4. 
>> at java.lang.Object.wait(Object.java)
>> 5. 
>> at java.lang.Object.wait(Object.java:485)
>> 6. 
>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
>> (GenericObjectPool.java:1104)
>> 7. 
>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
>> GenericObjectPool$Latch)
>> 8. 
>> at org.apache.commons.dbcp.PoolingDataSource.getConnection
>> (PoolingDataSource.java:106)
>> 9. 
>> at org.apache.commons.dbcp.BasicDataSource.getConnection
>> (BasicDataSource.java:1044)
>> 10. 
>> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
>> getConnection(ConnectionHelper.java:439)
>> 
>> 
>> 
>> 
>> 
>> 
>> You should be able to get thread dump any time unless JVM process is 
>> completely frozen IMO. Do you use jstack to get thread dump?
>> 
>> Marek
>> 
>> 
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d.hobi@gmx.ch>
>> Datum: 8. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>> 
>> "Hi Marek
>> 
>> Thanks for your suggestions!
>> 
>> Unfortunately, changing the JVM gc options so that the JVM will no longer 
>> hang just worked ONCE.
>> Therefore I was not able to get a "nice" stack trace anymore :-(
>> 
>> Connection Pool:
>> We had set the max connection pool size to 5. As you suggested I set it to
> 
>> 15. Same effect. Even 50 or 100 did not help.
>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is 
> this
>> a suspicious behavior or does it simply not need more connections and is 
>> therefore not spawning new connections?
>> 
>> Enabling debug, setting login and socket timeout on the postgres-jdbc-
> driver
>> did not help either.
>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
> core-
>> 2.6.0 to be "sure" it is not a pool bug.
>> Further we are not able to reproduce the hang with:
>> - Slower machine, postgres 8.3
>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
>> - Faster machine, mysql
>> 
>> This issue is slowly driving me nuts ;-)
>> 
>> Thanks!
>> Daniel
>> 
>> 
>> 
>> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
>> 
>>> Also randomly we had problem with not released connections in JBoss 
>>> connection pool and Postgres. Now long time no occurrence.
>>> In any case you can check number/list of open connections in psql . In 
> our
>> 
>>> case it was
>>> 97 open connection (by default 100 open connections in postgres and 3 are
> 
>>> reserved for admin). It happenned even if we had much smaller
>>> max connection pool size. As it happened randomly we did not find any 
>>> solution. We tried to change connection pool size and it does not
>>> happen any more. Still no idea what was cause of this. In any case using 
>>> psql you can find out easily if it is also your problem.
>>> 
>>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>>> Marek
>>> 
>>> -- 
>>> Marek Slama
>>> mslama@email.cz
>>> 
>>> 
>>> 
>>> ---------- Původní zpráva ----------
>>> Od: mslama@email.cz
>>> Datum: 7. 3. 2013
>>> Předmět: Re: JVM hangs during startup (indexing) (2)
>>> 
>>> "Quick look: It hangs in connection pool - connection pool if it cannot 
>>> return free connection to caller it waits forever by default.
>>> How big is your connection pool? (Maximum size? It would be good to patch
> 
>>> connection pool to see who/how many connections is
>>> used and why connections are used not release. In our config we use 
>>> connection max pool size 15 and it is enough.) So first
>>> fast fix would be to increase max connection pool size but if there is 
>>> problem with not released connection it will not help for long.
>>> 
>>> Good news is it is not problem with memory or gc (at least IMO).
>>> 
>>> Marek
>>> 
>>> -- 
>>> Marek Slama
>>> mslama@email.cz
>>> 
>>> 
>>> 
>>> ---------- Původní zpráva ----------
>>> Od: Daniel Hobi <d.hobi@gmx.ch>
>>> Datum: 7. 3. 2013
>>> Předmět: JVM hangs during startup (indexing) (2)
>>> 
>>> "Hi there
>>> 
>>> 
>>> 
>>> Some updates on our issue started here:
>>> 
>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
>> 201
>>> 30115135117.186960%40gmx.net%3E
>>> 
>>> 
>>> 
>>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>>> 
>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>>> 
>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>>> OpenJDK.
>>> 
>>> ...we increased the JVM heap size (up to 1.5gb).
>>> 
>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) 
> available.
>>> 
>>> 
>>> 
>>> But there is good news too:
>>> 
>>> - I've just played around with the JVM's garbage collection
>>> options. Even if the index process still hangs the JVM will no longer 
>> block.
>>> I was able to get an interesting thread dump:
>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>>> 
>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>>> hang occur faster (less than 100 minor gcs)
>>> 
>>> 
>>> 
>>> I would be very grateful if someone could have a look at this. We're 
>> running
>>> (slowly but surely) out of ideas :-(
>>> 
>>> 
>>> 
>>> Thanks!
>>> 
>>> Daniel""""


Mime
View raw message