continuum-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Spencer <pau...@apache.org>
Subject Re: Getting "java.lang.OutOfMemoryError: unable to create new native thread"
Date Fri, 13 Jun 2008 16:10:30 GMT
Emmanuel Venisse wrote:
> For each scheduler defined in Continuum, we start a quartz scheduler. We
> don't have the hand of the thread used internally by quartz.
> 

Each Quartz scheduler has is a configured number of worker threads, 15 
in this case.  At one time the number of worker threads was configured 
in components.xml, but that changed around 16-may-2006[1].  I have not 
found where the configuration was moved to.  With so many threads per 
Continiuum scheduler, any system is bound to run out of threads. I just 
hit the limit early.  I also suspect 15 work threads per Continuum 
schedule is just a waste.

[1]http://svn.apache.org/viewvc/continuum/trunk/continuum-core/src/main/resources/META-INF/plexus/components.xml?r1=344177&r2=406895&diff_format=h

> Emmanuel
> 
> On Fri, Jun 13, 2008 at 4:22 PM, Paul Spencer <paulsp@apache.org> wrote:
> 
>> Emmanuel Venisse wrote:
>>
>>> Can you send the list?
>>>
>> Below is the list of threads from jdp.  Continuum is the first application
>> to start followed by Archiva.
>>
>>  threads
>>> Group system:
>>>  (java.lang.ref.Reference$ReferenceHandler)0x12a9
>>>                   Reference Handler
>>>  cond. waiting
>>>  (java.lang.ref.Finalizer$FinalizerThread)0x12aa
>>>                  Finalizer
>>>  cond. waiting
>>>  (java.lang.Thread)0x12ab
>>>                   Signal Dispatcher
>>>  running
>>> Group main:
>>>  (java.lang.Thread)0x1
>>>                  main
>>> running
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ae
>>>                   defaultScheduler_Worker-0
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12af
>>>                   defaultScheduler_Worker-1
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b0
>>>                   defaultScheduler_Worker-2
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b1
>>>                   defaultScheduler_Worker-3
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b2
>>>                   defaultScheduler_Worker-4
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b3
>>>                   defaultScheduler_Worker-5
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b4
>>>                   defaultScheduler_Worker-6
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b5
>>>                   defaultScheduler_Worker-7
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b6
>>>                   defaultScheduler_Worker-8
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b7
>>>                   defaultScheduler_Worker-9
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b8
>>>                   defaultScheduler_Worker-10
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b9
>>>                   defaultScheduler_Worker-11
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ba
>>>                   defaultScheduler_Worker-12
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12bb
>>>                   defaultScheduler_Worker-13
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12bc
>>>                   defaultScheduler_Worker-14
>>> cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12bd
>>> Thread-2                                               cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12be
>>> Thread-3                                               cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12bf
>>> Thread-4                                               cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12c0
>>> Thread-5                                               cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12c1
>>> Thread-6                                               cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c2
>>>                   defaultScheduler_Worker-0
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c3
>>>                   defaultScheduler_Worker-1
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c4
>>>                   defaultScheduler_Worker-2
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c5
>>>                   defaultScheduler_Worker-3
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c6
>>>                   defaultScheduler_Worker-4
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c7
>>>                   defaultScheduler_Worker-5
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c8
>>>                   defaultScheduler_Worker-6
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c9
>>>                   defaultScheduler_Worker-7
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ca
>>>                   defaultScheduler_Worker-8
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cb
>>>                   defaultScheduler_Worker-9
>>>  cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cc
>>>                   defaultScheduler_Worker-10
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cd
>>>                   defaultScheduler_Worker-11
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ce
>>>                   defaultScheduler_Worker-12
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cf
>>>                   defaultScheduler_Worker-13
>>> cond. waiting
>>>  (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12d0
>>>                   defaultScheduler_Worker-14
>>> cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12d1
>>> Thread-7                                               cond. waiting
>>>  (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12d2
>>> Thread-8                                               cond. waiting
>>>  (java.lang.Thread)0x12d3
>>>                   ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>> sleeping
>>>  (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d4
>>>                  TP-Processor1
>>>  cond. waiting
>>>  (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d5
>>>                  TP-Processor2
>>>  cond. waiting
>>>  (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d6
>>>                  TP-Processor3
>>>  cond. waiting
>>>  (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d7
>>>                  TP-Processor4
>>>  running
>>>  (java.lang.Thread)0x12d8
>>>                   TP-Monitor
>>> cond. waiting
>>> Group QuartzScheduler:defaultScheduler:
>>>  (org.quartz.core.QuartzSchedulerThread)0x12db
>>>                  defaultScheduler_QuartzSchedulerThread
>>> sleeping
>>> Group QuartzScheduler:defaultScheduler:
>>>  (org.quartz.core.QuartzSchedulerThread)0x12dc
>>>                  defaultScheduler_QuartzSchedulerThread
>>> sleeping
>>> Group SeedGenerator ThreadGroup:
>>>  (sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)0x12dd
>>>              Noisy Thread                                           cond.
>>> waiting
>>>  (java.lang.Thread)0x12de
>>>                   SeedGenerator Thread
>>> cond. waiting
>>>
>> Looks like a lot of worker threads allocated to Quartz for BOTH Continuum
>> and Archiva.
>>
>> 1) For Continuum, are 15 Quartz worker threads really need for a small
>> system, less then 10 builds a day and none concurrently?
>>
>> 2) Does Archiva really need 15 Quartz worker threads?
>>
>> 3) Are the Quartz worker thread counts configurable?  If so, how?
>>
>>
>>
>>> On Thu, Jun 12, 2008 at 7:14 PM, Paul Spencer <paulsp@apache.org> wrote:
>>>
>>>  Emmanuel Venisse wrote:
>>>>  I'm sure you'll consume less threads if you use an external db instead
>>>>> of
>>>>> the embedded.
>>>>>
>>>>>  I have done this and it has allowed both application run, although the
>>>> thread count is 59.  Continuum consumes 21 threads and Archiva consumes
>>>> 20
>>>> threads in the Tomcat server.
>>>>
>>>> 21 threads appears to be a large number of threads.  Is this tuneable?
>>>>
>>>>
>>>>
>>>>  Emmanuel
>>>>> On Tue, Jun 10, 2008 at 12:27 AM, Paul Spencer <paulsp@apache.org>
>>>>> wrote:
>>>>>
>>>>>  I think I have found the cause of this error.  The Continuum web
>>>>>
>>>>>> application and Tomcat are consuming 44 threads when all is idle.
 The
>>>>>> default max thread count on HP-UX is 64 per process. When Archiva
is
>>>>>> added
>>>>>> to the Tomcat instance the thread count hits 64.
>>>>>>
>>>>>> Below is the database configuration in context.xml.
>>>>>>  <Resource name="jdbc/continuum"
>>>>>>         auth="Container"
>>>>>>         type="javax.sql.DataSource"
>>>>>>         username="sa"
>>>>>>         password=""
>>>>>>         driverClassName="org.apache.derby.jdbc.EmbeddedDriver"
>>>>>>         url="jdbc:derby:/internal/continuum/db/continuum;create=true"
>>>>>>  />
>>>>>>
>>>>>> So my questions are:
>>>>>> 1) Why are so many threads being used?
>>>>>>
>>>>>> 2) How can be minimize the thread count?
>>>>>>
>>>>>> Paul Spencer
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Paul Spencer wrote:
>>>>>>
>>>>>>  Wendy Smoak wrote:
>>>>>>
>>>>>>>  On Mon, Jun 9, 2008 at 8:52 AM, Paul Spencer <paulsp@apache.org>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  Now that I have Archiva and Continuum running in the same
Tomcat
>>>>>>>> 6.0.16,
>>>>>>>>
>>>>>>>>  I
>>>>>>>>> am seeing the exception "java.lang.OutOfMemoryError:
unable to
>>>>>>>>> create
>>>>>>>>> new
>>>>>>>>> native thread" in catalina.out.
>>>>>>>>>
>>>>>>>>> JVM = build 1.5.0.01
>>>>>>>>> OS = HP-UX 11.11
>>>>>>>>>
>>>>>>>>>  That looks familiar. :)  Who is the JDK vendor?  (ISTR
that
>>>>>>>>> something
>>>>>>>>>
>>>>>>>> in Redback requires a Sun JDK...)
>>>>>>>>
>>>>>>>>  $ $JAVA_HOME/bin/java -version
>>>>>>>>
>>>>>>> java version "1.5.0.01" Java(TM) 2 Runtime Environment, Standard
>>>>>>> Edition
>>>>>>> (build 1.5.0.01-_06_jun_2005_05_20)
>>>>>>> Java HotSpot(TM) Server VM (build 1.5.0.01 jinteg:06.06.05-04:39
>>>>>>> PA2.0
>>>>>>> (aCC_AP), mixed mode)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>
>>>>>>>>  I suspect the memory and/or stack configuration need to
be altered,
>>>>>>>> but
>>>>>>>>
>>>>>>>>  I am
>>>>>>>>> not sure which ones to alter.  Currently CATALINA_OPT
=
>>>>>>>>> '-Dapplication.base=$ARCHIVA_BASE -Dapplication.home=$ARCHIVA_BASE"
>>>>>>>>>
>>>>>>>>>  If the message is in the Catalina log, you probably
need to give
>>>>>>>>>
>>>>>>>> Tomcat itself more memory.  This might help:
>>>>>>>> http://wiki.apache.org/tomcat/FAQ/Memory
>>>>>>>>
>>>>>>>>
>>>>>>>>  Paul Spencer
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  Paul Spencer
>>>>
> 


Mime
View raw message