tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pid <...@pidster.com>
Subject Re: jvm exits without trace
Date Fri, 05 Mar 2010 09:09:36 GMT
On 05/03/2010 08:41, Taylan Develioglu wrote:
> Pid, that would assume we had a working<  1.6.10 version before that we
> replaced.

That it would.

> We've run 1.6.10 upwards succesfully for a very long time. So I don't
> see the point in doing this.

I must have missed that.

How is the HTTP connector performing?


p

> On Wed, 2010-03-03 at 12:00 +0100, Pid wrote:
>> On 03/03/2010 09:11, Taylan Develioglu wrote:
>>> Downgrading to 1.6.0_16 did not help. I'm replacing the apr connector
>>> with http now.
>>
>> As Chuck mentioned in the other thread, significant changes occurred at
>> 1.6.10, so trying the release before (1.6.7) might be necessary to
>> establish a better determination.
>>
>>
>> p
>>
>>> On Wed, 2010-02-24 at 14:52 +0100, Carl wrote:
>>>> Taylan,
>>>>
>>>>> The failures we've seen are in anywhere between 8 hours to a week of
>>>>> runtime.
>>>>
>>>> The timing of the failures seems similar.
>>>>
>>>>> We have also had failures with hotspot error files (hs_err) present,
and
>>>>> the cause specified was indeed SIGSEGV indicating a page fault.
>>>>
>>>> I have never seen any hs_* files but have seen core files where strace
>>>> showed the jvm stopped on a seg fault.
>>>>
>>>>> We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when
>>>>> the situation allows (during regular updates of the application, or a
>>>>> crash) to see if that helps.
>>>>
>>>> I have used jdk 1.6.0_17 and 1.6.0_18 with the same results... have not
>>>> tried 1.6.0_16.  Please post your results of this trial.
>>>>
>>>>> Running tomcat on the
>>>>> foreground might show something, but then again I could be waiting for
a
>>>>> month for it to happen.
>>>>
>>>> Yes, this has been part of my problem as anytime we change something, we
>>>> have to wait a week for the server to fail.
>>>>
>>>> In one sense, I am fortunate that I have a little more flexibility than you.
>>>> I have two servers (different hardware) but only need one in service at a
>>>> time.  Therefore, I always have one server I can test ideas on although I
>>>> have never been able to develop a meaningful stress test, i.e., the only
way
>>>> I can test a change is to put it in production.
>>>>
>>>> Thanks,
>>>>
>>>> Carl
>>>>
>>>> ----- Original Message -----
>>>> From: "Taylan Develioglu"<tdevelioglu@ebuddy.com>
>>>> To: "Tomcat Users List"<users@tomcat.apache.org>
>>>> Sent: Wednesday, February 24, 2010 8:31 AM
>>>> Subject: Re: jvm exits without trace
>>>>
>>>>
>>>>> Hello Carl,
>>>>>
>>>>> The failures we've seen are in anywhere between 8 hours to a week of
>>>>> runtime. Most of them have (still) been running for almost a month
>>>>> without failure. There are ~100 machines.
>>>>>
>>>>>>  From the top of my head, I think we've had about 10+ failures now.
>>>>>
>>>>> We have also had failures with hotspot error files (hs_err) present,
and
>>>>> the cause specified was indeed SIGSEGV indicating a page fault. But I
>>>>> don't know if the two are related.
>>>>>
>>>>> We also use jdk 1.6.0_18, I'm downgrading the machines to 1.6.0_16 when
>>>>> the situation allows (during regular updates of the application, or a
>>>>> crash) to see if that helps.
>>>>>
>>>>> It might be useful to note that the failures happen with tomcat 6.0.20
>>>>> as well as 6.0.24.
>>>>>
>>>>> As far as load concerns, I haven't had a failure on an idle machines.
>>>>> The machines are well loaded, but only at a fraction limit in regards
to
>>>>> load and cpu utilization.
>>>>> Most memory is commited to tomcat, where a 24G machine would have 18G
>>>>> allocated to heap, 128M to permgen and some unspecified amount would
get
>>>>> used by jni for apr. About 4G remains free after calculating taking into
>>>>> account the jvm itsself.
>>>>> A 16G machine would have 12G allocated to the heap.
>>>>>
>>>>> Besides the fact that our apps heavily use nio and mina I wouldn't say
>>>>> there's anything else noteworthy. There can be anywhere up to 10000
>>>>> concurrents on one machine.
>>>>>
>>>>> I had searched for coredumps, but no luck. Running tomcat on the
>>>>> foreground might show something, but then again I could be waiting for
a
>>>>> month for it to happen.
>>>>>
>>>>> On Wed, 2010-02-24 at 12:42 +0100, Carl wrote:
>>>>>> Taylan,
>>>>>>
>>>>>> I am the person who started the "Tomcat dies suddenly" thread which
I
>>>>>> still
>>>>>> haven't resolved.  I am curious about the pattern of failures you
are
>>>>>> experiencing because they may provide some clues to my problem. 
In my
>>>>>> case,
>>>>>> the system will run for 15 minutes to 10 days before failing (most
of the
>>>>>> time it is several days to a week.)  It appears to die from a seg
fault
>>>>>> in
>>>>>> the JVM (I am using Sun 1.6.0_18 but have tried previous versions)...
you
>>>>>> may be able to see the cause of the failure from the core file (the
core
>>>>>> files on my systems were in several directories so you may have to
do a
>>>>>> 'find' to locate them.)  Load may be a factor but the failures generally
>>>>>> come after the load has been heavy for a while.  I am running a couple
of
>>>>>> applications and it seems the failures are more frequent when people
are
>>>>>> hitting the additional apps (the primary app is always used, the
>>>>>> remaining
>>>>>> apps are used sporatically.)
>>>>>>
>>>>>> How does this compare to what you are experiencing?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Carl
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Taylan Develioglu"<tdevelioglu@ebuddy.com>
>>>>>> To: "Tomcat Users List"<users@tomcat.apache.org>;<pid@pidster.com>
>>>>>> Sent: Wednesday, February 24, 2010 5:09 AM
>>>>>> Subject: Re: jvm exits without trace
>>>>>>
>>>>>>
>>>>>>> The GC log shows plenty of heap space left in all the spaces.
>>>>>>>
>>>>>>> I purposely didn't bother replacing the variables because I figured
>>>>>>> they
>>>>>>> would not be relevant.
>>>>>>>
>>>>>>> But if you think they might provide clues they're as follows:
>>>>>>>
>>>>>>> JAVA_HEAP_SIZE=18432M
>>>>>>> JAVA_EDEN_SIZE=$(($(echo $JAVA_HEAP_SIZE|sed 's/M$\|G$//')/6))M
>>>>>>> JAVA_PERM_SIZE=128M
>>>>>>> JAVA_STCK_SIZE=128K
>>>>>>>
>>>>>>> EDEN_SIZE is 1/6th of total heap.
>>>>>>>
>>>>>>> And I said there was nothing in the system logs.
>>>>>>> But you get a couple of points for trying.
>>>>>>>
>>>>>>> On Wed, 2010-02-24 at 10:44 +0100, Pid wrote:
>>>>>>>> On 24/02/2010 09:36, Taylan Develioglu wrote:
>>>>>>>>> I thought I'd add the connector definitions too, :
>>>>>>>>>
>>>>>>>>>       <Connector port="80"
>>>>>>>>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>>>>>>>>>                   compression="1024" keepAliveTimeout="60000"
>>>>>>>>> maxKeepAliveRequests="-1"
>>>>>>>>>                   enableLookups="false" redirectPort="443"
>>>>>>>>> maxThreads="150"
>>>>>>>>> pollerSize="32768"
>>>>>>>>>                   pollerThreadCount="4"/>
>>>>>>>>>
>>>>>>>>>        <Connector port="443"
>>>>>>>>> protocol="org.apache.coyote.http11.Http11AprProtocol"
>>>>>>>>> SSLEnabled="true"
>>>>>>>>>                   enableLookups="false" maxThreads="10"
scheme="https"
>>>>>>>>> secure="true"
>>>>>>>>>                   SSLCertificateFile="/etc/ssl/private/something.crt"
>>>>>>>>>
>>>>>>>>> SSLCertificateKeyFile="/etc/ssl/private/something.key"
>>>>>>>>>                   SSLCACertificateFile="/etc/ssl/certs/ca.crt"/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 2010-02-24 at 10:23 +0100, Taylan Develioglu
wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I have jvm's, running tomcat and our application,
exiting
>>>>>>>>>> mysteriously,
>>>>>>>>>> and was wondering if anyone could give me some advice
on how to
>>>>>>>>>> debug
>>>>>>>>>> this thing.
>>>>>>>>>>
>>>>>>>>>> There is nothing in catalina.out, nor our application
logs, and no
>>>>>>>>>> hotspot error file. GC log looks normal. No trace
in system logs.
>>>>>>>>>>
>>>>>>>>>> I am left completely clueless :(, has anyone dealt
with a problem
>>>>>>>>>> like
>>>>>>>>>> this before?
>>>>>>>>>>
>>>>>>>>>> Any help appreciated.
>>>>>>>>>>
>>>>>>>>>> - Tomcat 6.0.24
>>>>>>>>>> - TC native 1.1.18
>>>>>>>>>> - APR 1.3.9
>>>>>>>>>> - Sun JDK 6u18
>>>>>>>>>> - Debian Lenny, 2.6.31.10-amd64
>>>>>>>>>>
>>>>>>>>>> 2 servlets, one as ROOT. 2 HTTP connectors that use
TCNative/APR.
>>>>>>>>>>
>>>>>>>>>> JAVA_OPTS ( ):
>>>>>>>>>>
>>>>>>>>>>        -verbose:gc
>>>>>>>>>>        -Djava.awt.headless=true
>>>>>>>>>>        -Dsun.net.inetaddr.ttl=60
>>>>>>>>>>        -Dfile.encoding=UTF-8
>>>>>>>>>>        -Djava.io.tmpdir=$TMP_DIR
>>>>>>>>>>        -Djava.library.path=/usr/local/lib
>>>>>>>>>>        -Djava.endorsed.dirs=$CATALINA_BASE/endorsed
>>>>>>>>>>        -Dcatalina.base=$CATALINA_BASE
>>>>>>>>>>        -Dcatalina.home=$CATALINA_HOME
>>>>>>>>
>>>>>>>>>>      -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>>>>>> -Djava.util.logging.config.file="$CATALINA_BASE/conf/logging.properties"
>>>>>>>>>>        -XX:+PrintGCDetails
>>>>>>>>>>        -Xloggc:$CATALINA_BASE/logs/gc.log
>>>>>>>>>>        -XX:+UseConcMarkSweepGC
>>>>>>>>>>        -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>>>        -Xms$JAVA_HEAP_SIZE
>>>>>>>>>>        -Xmx$JAVA_HEAP_SIZE
>>>>>>>>>>        -XX:NewSize=$JAVA_EDEN_SIZE
>>>>>>>>>>        -XX:MaxNewSize=$JAVA_EDEN_SIZE
>>>>>>>>>>        -XX:PermSize=$JAVA_PERM_SIZE
>>>>>>>>>>        -XX:MaxPermSize=$JAVA_PERM_SIZE
>>>>>>>>>>        -Xss$JAVA_STCK_SIZE
>>>>>>>>>>        -XX:+UseLargePages
>>>>>>>>
>>>>>>>> There's no actual heap size settings in the above.  But you
get a
>>>>>>>> couple
>>>>>>>> of points for trying.
>>>>>>>>
>>>>>>>> Google "Linux Out Of Memory killer" or "OOM Killer" and then
check the
>>>>>>>> server logs carefully.  (e.g. /var/log/messages)
>>>>>>>>
>>>>>>>>
>>>>>>>> p
>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message