jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <msl...@email.cz>
Subject Re: Re: JVM hangs during startup (indexing) (2)
Date Fri, 08 Mar 2013 09:49:09 GMT
It is really weird that JVM process is frozen and you cannot get thread 
dump. I never saw such problem. It looks like HW or OS problem.
Does it happen also on another machine? From your log I also see you use 
OpenJDK. Try latest Oracle JDK to see if it makes any difference.




If you monitor frozen JVM process is there any activity? (CPU/io). Is there 
any OS limit on process mem size? Check ulimit settings. (ulimit -a)




Marek


"Hi Marek

I use jstack -F <pid> to get thread dumps.
- Except once the output of jstack is the only one I get (= the first dump I
provided) 
- Once (the jvm was not completely frozen, just the indexing made no further
progress) jstack also made a dump in our log file (= second dump)

Another result after killing the jvm process with "kill -4 <pid>" (kill -3 
does not work due to a frozen jvm)
http://pastebin.com/qc2bUkbk

Daniel

Am 08.03.2013 um 10:16 schrieb <mslama@email.cz>:

> I checked 2 thread dumps you provided so I can only comment what I saw 
> there. I did not see any deadlock. Only
> problem I saw was in main thread where it was waiting for connection.
> 
> 
> 
> 
> 
> 1. 
> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x
> 00007fefd3812000]
> 2. 
> java.lang.Thread.State: WAITING (on object monitor)
> 3. 
> at java.lang.Object.$$YJP$$wait(Native Method)
> 4. 
> at java.lang.Object.wait(Object.java)
> 5. 
> at java.lang.Object.wait(Object.java:485)
> 6. 
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject
> (GenericObjectPool.java:1104)
> 7. 
> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl.
> GenericObjectPool$Latch)
> 8. 
> at org.apache.commons.dbcp.PoolingDataSource.getConnection
> (PoolingDataSource.java:106)
> 9. 
> at org.apache.commons.dbcp.BasicDataSource.getConnection
> (BasicDataSource.java:1044)
> 10. 
> at org.apache.jackrabbit.core.util.db.ConnectionHelper.
> getConnection(ConnectionHelper.java:439)
> 
> 
> 
> 
> 
> 
> You should be able to get thread dump any time unless JVM process is 
> completely frozen IMO. Do you use jstack to get thread dump?
> 
> Marek
> 
> 
> ---------- Původní zpráva ----------
> Od: Daniel Hobi <d.hobi@gmx.ch>
> Datum: 8. 3. 2013
> Předmět: Re: JVM hangs during startup (indexing) (2)
> 
> "Hi Marek
> 
> Thanks for your suggestions!
> 
> Unfortunately, changing the JVM gc options so that the JVM will no longer 
> hang just worked ONCE.
> Therefore I was not able to get a "nice" stack trace anymore :-(
> 
> Connection Pool:
> We had set the max connection pool size to 5. As you suggested I set it to

> 15. Same effect. Even 50 or 100 did not help.
> Interestingly, postgres shows just 3 open connection for jackrabbit. Is 
this
> a suspicious behavior or does it simply not need more connections and is 
> therefore not spawning new connections?
> 
> Enabling debug, setting login and socket timeout on the postgres-jdbc-
driver
> did not help either.
> I even changed the pool implementation from dbcp to c3p0 in jackrabbit-
core-
> 2.6.0 to be "sure" it is not a pool bug.
> Further we are not able to reproduce the hang with:
> - Slower machine, postgres 8.3
> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far)
> - Faster machine, mysql
> 
> This issue is slowly driving me nuts ;-)
> 
> Thanks!
> Daniel
> 
> 
> 
> Am 07.03.2013 um 16:24 schrieb mslama@email.cz:
> 
>> Also randomly we had problem with not released connections in JBoss 
>> connection pool and Postgres. Now long time no occurrence.
>> In any case you can check number/list of open connections in psql . In 
our
> 
>> case it was
>> 97 open connection (by default 100 open connections in postgres and 3 are

>> reserved for admin). It happenned even if we had much smaller
>> max connection pool size. As it happened randomly we did not find any 
>> solution. We tried to change connection pool size and it does not
>> happen any more. Still no idea what was cause of this. In any case using 
>> psql you can find out easily if it is also your problem.
>> 
>> Query is for 9.x: SELECT * FROM pg_stat_activity;
>> Marek
>> 
>> -- 
>> Marek Slama
>> mslama@email.cz
>> 
>> 
>> 
>> ---------- Původní zpráva ----------
>> Od: mslama@email.cz
>> Datum: 7. 3. 2013
>> Předmět: Re: JVM hangs during startup (indexing) (2)
>> 
>> "Quick look: It hangs in connection pool - connection pool if it cannot 
>> return free connection to caller it waits forever by default.
>> How big is your connection pool? (Maximum size? It would be good to patch

>> connection pool to see who/how many connections is
>> used and why connections are used not release. In our config we use 
>> connection max pool size 15 and it is enough.) So first
>> fast fix would be to increase max connection pool size but if there is 
>> problem with not released connection it will not help for long.
>> 
>> Good news is it is not problem with memory or gc (at least IMO).
>> 
>> Marek
>> 
>> -- 
>> Marek Slama
>> mslama@email.cz
>> 
>> 
>> 
>> ---------- Původní zpráva ----------
>> Od: Daniel Hobi <d.hobi@gmx.ch>
>> Datum: 7. 3. 2013
>> Předmět: JVM hangs during startup (indexing) (2)
>> 
>> "Hi there
>> 
>> 
>> 
>> Some updates on our issue started here:
>> 
>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C
> 201
>> 30115135117.186960%40gmx.net%3E
>> 
>> 
>> 
>> We're still facing the JVM freeze (possibly during minor gcs) even if...
>> 
>> ...we updated Jackrabbit from 2.2.13 to 2.6.0.
>> 
>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried
>> OpenJDK.
>> 
>> ...we increased the JVM heap size (up to 1.5gb).
>> 
>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) 
available.
>> 
>> 
>> 
>> But there is good news too:
>> 
>> - I've just played around with the JVM's garbage collection
>> options. Even if the index process still hangs the JVM will no longer 
> block.
>> I was able to get an interesting thread dump:
>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8
>> 
>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the
>> hang occur faster (less than 100 minor gcs)
>> 
>> 
>> 
>> I would be very grateful if someone could have a look at this. We're 
> running
>> (slowly but surely) out of ideas :-(
>> 
>> 
>> 
>> Thanks!
>> 
>> Daniel""""
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message