From users-return-6589-apmail-continuum-users-archive=continuum.apache.org@continuum.apache.org Fri Jun 13 16:24:04 2008 Return-Path: Delivered-To: apmail-continuum-users-archive@www.apache.org Received: (qmail 28076 invoked from network); 13 Jun 2008 16:24:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Jun 2008 16:24:03 -0000 Received: (qmail 54126 invoked by uid 500); 13 Jun 2008 16:24:05 -0000 Delivered-To: apmail-continuum-users-archive@continuum.apache.org Received: (qmail 54095 invoked by uid 500); 13 Jun 2008 16:24:05 -0000 Mailing-List: contact users-help@continuum.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@continuum.apache.org Delivered-To: mailing list users@continuum.apache.org Received: (qmail 54084 invoked by uid 99); 13 Jun 2008 16:24:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jun 2008 09:24:04 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of emmanuel.venisse@gmail.com designates 66.249.82.229 as permitted sender) Received: from [66.249.82.229] (HELO wx-out-0506.google.com) (66.249.82.229) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jun 2008 16:23:16 +0000 Received: by wx-out-0506.google.com with SMTP id h30so1650746wxd.14 for ; Fri, 13 Jun 2008 09:23:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=5Gauiic0OY+UCu8d24S2Uon5/4c03xOn3aSmlhXyU18=; b=R8ZFLTxQ0MiY2BfwWPNQbX6TL16cip79vNifhliIl5OXlcp/QPzmAmvetM5t/iQ3sV 2NXJBDWNIBAaw6quUH2ncXN541Z+TtXn1V+f1wAMkQGe3KKWPz4H5/qyECV5Qej+LmG7 P+kg1mCeW56Dkk7PkyZv75HH40v05/fe0EX9I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=D0zBBj7Avzunqo88H5WTck+3cqcQfF8F7F1JIuoE4DZEp7VTD1Sxb/FkUWL+P+RLQB qi6rKy8bKrYiCpEIyDXQxPa8CqzXaNhPDdF3l1ZsMWeRn8O792tNbFkTYofz58Or37l+ RklU8ezpgdYObb1/dNgygKBPYd6ArttUvbaoM= Received: by 10.70.43.16 with SMTP id q16mr3166118wxq.82.1213374213331; Fri, 13 Jun 2008 09:23:33 -0700 (PDT) Received: by 10.70.72.4 with HTTP; Fri, 13 Jun 2008 09:23:32 -0700 (PDT) Message-ID: Date: Fri, 13 Jun 2008 18:23:32 +0200 From: "Emmanuel Venisse" To: users@continuum.apache.org Subject: Re: Getting "java.lang.OutOfMemoryError: unable to create new native thread" In-Reply-To: <48529BF6.1060109@apache.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_9510_31853450.1213374212739" References: <484D51BC.8030506@apache.org> <484D578F.6030109@apache.org> <484DAE57.3060204@apache.org> <4851596A.20303@apache.org> <485282B9.3010903@apache.org> <48529BF6.1060109@apache.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_9510_31853450.1213374212739 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hmm, it seems to be an error in the merge. You can copy the org.apache.maven.continuum.scheduler.ContinuumScheduler component descriptor in your application.xml It should work because the continuum scheduler always read properties. Can you file an issue about it, so I'll look at it in details? Thanks for your collaboration; Emmanuel On Fri, Jun 13, 2008 at 6:10 PM, Paul Spencer wrote: > Emmanuel Venisse wrote: > >> For each scheduler defined in Continuum, we start a quartz scheduler. We >> don't have the hand of the thread used internally by quartz. >> >> > Each Quartz scheduler has is a configured number of worker threads, 15 in > this case. At one time the number of worker threads was configured in > components.xml, but that changed around 16-may-2006[1]. I have not found > where the configuration was moved to. With so many threads per Continiuum > scheduler, any system is bound to run out of threads. I just hit the limit > early. I also suspect 15 work threads per Continuum schedule is just a > waste. > > [1] > http://svn.apache.org/viewvc/continuum/trunk/continuum-core/src/main/resources/META-INF/plexus/components.xml?r1=344177&r2=406895&diff_format=h > > > Emmanuel >> >> On Fri, Jun 13, 2008 at 4:22 PM, Paul Spencer wrote: >> >> Emmanuel Venisse wrote: >>> >>> Can you send the list? >>>> >>>> Below is the list of threads from jdp. Continuum is the first >>> application >>> to start followed by Archiva. >>> >>> threads >>> >>>> Group system: >>>> (java.lang.ref.Reference$ReferenceHandler)0x12a9 >>>> Reference Handler >>>> cond. waiting >>>> (java.lang.ref.Finalizer$FinalizerThread)0x12aa >>>> Finalizer >>>> cond. waiting >>>> (java.lang.Thread)0x12ab >>>> Signal Dispatcher >>>> running >>>> Group main: >>>> (java.lang.Thread)0x1 >>>> main >>>> running >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ae >>>> defaultScheduler_Worker-0 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12af >>>> defaultScheduler_Worker-1 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b0 >>>> defaultScheduler_Worker-2 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b1 >>>> defaultScheduler_Worker-3 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b2 >>>> defaultScheduler_Worker-4 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b3 >>>> defaultScheduler_Worker-5 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b4 >>>> defaultScheduler_Worker-6 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b5 >>>> defaultScheduler_Worker-7 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b6 >>>> defaultScheduler_Worker-8 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b7 >>>> defaultScheduler_Worker-9 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b8 >>>> defaultScheduler_Worker-10 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12b9 >>>> defaultScheduler_Worker-11 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ba >>>> defaultScheduler_Worker-12 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12bb >>>> defaultScheduler_Worker-13 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12bc >>>> defaultScheduler_Worker-14 >>>> cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12bd >>>> Thread-2 cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12be >>>> Thread-3 cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12bf >>>> Thread-4 cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12c0 >>>> Thread-5 cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12c1 >>>> Thread-6 cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c2 >>>> defaultScheduler_Worker-0 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c3 >>>> defaultScheduler_Worker-1 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c4 >>>> defaultScheduler_Worker-2 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c5 >>>> defaultScheduler_Worker-3 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c6 >>>> defaultScheduler_Worker-4 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c7 >>>> defaultScheduler_Worker-5 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c8 >>>> defaultScheduler_Worker-6 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12c9 >>>> defaultScheduler_Worker-7 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ca >>>> defaultScheduler_Worker-8 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cb >>>> defaultScheduler_Worker-9 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cc >>>> defaultScheduler_Worker-10 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cd >>>> defaultScheduler_Worker-11 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12ce >>>> defaultScheduler_Worker-12 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12cf >>>> defaultScheduler_Worker-13 >>>> cond. waiting >>>> (org.quartz.simpl.SimpleThreadPool$WorkerThread)0x12d0 >>>> defaultScheduler_Worker-14 >>>> cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12d1 >>>> Thread-7 cond. waiting >>>> >>>> (org.codehaus.plexus.taskqueue.execution.ThreadedTaskQueueExecutor$ExecutorRunnable)0x12d2 >>>> Thread-8 cond. waiting >>>> (java.lang.Thread)0x12d3 >>>> ContainerBackgroundProcessor[StandardEngine[Catalina]] >>>> sleeping >>>> (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d4 >>>> TP-Processor1 >>>> cond. waiting >>>> (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d5 >>>> TP-Processor2 >>>> cond. waiting >>>> (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d6 >>>> TP-Processor3 >>>> cond. waiting >>>> (org.apache.tomcat.util.threads.ThreadWithAttributes)0x12d7 >>>> TP-Processor4 >>>> running >>>> (java.lang.Thread)0x12d8 >>>> TP-Monitor >>>> cond. waiting >>>> Group QuartzScheduler:defaultScheduler: >>>> (org.quartz.core.QuartzSchedulerThread)0x12db >>>> defaultScheduler_QuartzSchedulerThread >>>> sleeping >>>> Group QuartzScheduler:defaultScheduler: >>>> (org.quartz.core.QuartzSchedulerThread)0x12dc >>>> defaultScheduler_QuartzSchedulerThread >>>> sleeping >>>> Group SeedGenerator ThreadGroup: >>>> >>>> (sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)0x12dd >>>> Noisy Thread cond. >>>> waiting >>>> (java.lang.Thread)0x12de >>>> SeedGenerator Thread >>>> cond. waiting >>>> >>>> Looks like a lot of worker threads allocated to Quartz for BOTH >>> Continuum >>> and Archiva. >>> >>> 1) For Continuum, are 15 Quartz worker threads really need for a small >>> system, less then 10 builds a day and none concurrently? >>> >>> 2) Does Archiva really need 15 Quartz worker threads? >>> >>> 3) Are the Quartz worker thread counts configurable? If so, how? >>> >>> >>> >>> On Thu, Jun 12, 2008 at 7:14 PM, Paul Spencer >>>> wrote: >>>> >>>> Emmanuel Venisse wrote: >>>> >>>>> I'm sure you'll consume less threads if you use an external db instead >>>>> >>>>>> of >>>>>> the embedded. >>>>>> >>>>>> I have done this and it has allowed both application run, although >>>>>> the >>>>>> >>>>> thread count is 59. Continuum consumes 21 threads and Archiva consumes >>>>> 20 >>>>> threads in the Tomcat server. >>>>> >>>>> 21 threads appears to be a large number of threads. Is this tuneable? >>>>> >>>>> >>>>> >>>>> Emmanuel >>>>> >>>>>> On Tue, Jun 10, 2008 at 12:27 AM, Paul Spencer >>>>>> wrote: >>>>>> >>>>>> I think I have found the cause of this error. The Continuum web >>>>>> >>>>>> application and Tomcat are consuming 44 threads when all is idle. >>>>>>> The >>>>>>> default max thread count on HP-UX is 64 per process. When Archiva is >>>>>>> added >>>>>>> to the Tomcat instance the thread count hits 64. >>>>>>> >>>>>>> Below is the database configuration in context.xml. >>>>>>> >>>>>> auth="Container" >>>>>>> type="javax.sql.DataSource" >>>>>>> username="sa" >>>>>>> password="" >>>>>>> driverClassName="org.apache.derby.jdbc.EmbeddedDriver" >>>>>>> url="jdbc:derby:/internal/continuum/db/continuum;create=true" >>>>>>> /> >>>>>>> >>>>>>> So my questions are: >>>>>>> 1) Why are so many threads being used? >>>>>>> >>>>>>> 2) How can be minimize the thread count? >>>>>>> >>>>>>> Paul Spencer >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Paul Spencer wrote: >>>>>>> >>>>>>> Wendy Smoak wrote: >>>>>>> >>>>>>> On Mon, Jun 9, 2008 at 8:52 AM, Paul Spencer >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Now that I have Archiva and Continuum running in the same Tomcat >>>>>>>>> 6.0.16, >>>>>>>>> >>>>>>>>> I >>>>>>>>> >>>>>>>>>> am seeing the exception "java.lang.OutOfMemoryError: unable to >>>>>>>>>> create >>>>>>>>>> new >>>>>>>>>> native thread" in catalina.out. >>>>>>>>>> >>>>>>>>>> JVM = build 1.5.0.01 >>>>>>>>>> OS = HP-UX 11.11 >>>>>>>>>> >>>>>>>>>> That looks familiar. :) Who is the JDK vendor? (ISTR that >>>>>>>>>> something >>>>>>>>>> >>>>>>>>>> in Redback requires a Sun JDK...) >>>>>>>>> >>>>>>>>> $ $JAVA_HOME/bin/java -version >>>>>>>>> >>>>>>>>> java version "1.5.0.01" Java(TM) 2 Runtime Environment, Standard >>>>>>>> Edition >>>>>>>> (build 1.5.0.01-_06_jun_2005_05_20) >>>>>>>> Java HotSpot(TM) Server VM (build 1.5.0.01 jinteg:06.06.05-04:39 >>>>>>>> PA2.0 >>>>>>>> (aCC_AP), mixed mode) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>>>>> >>>>>>>> I suspect the memory and/or stack configuration need to be >>>>>>>>> altered, >>>>>>>>> but >>>>>>>>> >>>>>>>>> I am >>>>>>>>> >>>>>>>>>> not sure which ones to alter. Currently CATALINA_OPT = >>>>>>>>>> '-Dapplication.base=$ARCHIVA_BASE >>>>>>>>>> -Dapplication.home=$ARCHIVA_BASE" >>>>>>>>>> >>>>>>>>>> If the message is in the Catalina log, you probably need to give >>>>>>>>>> >>>>>>>>>> Tomcat itself more memory. This might help: >>>>>>>>> http://wiki.apache.org/tomcat/FAQ/Memory >>>>>>>>> >>>>>>>>> >>>>>>>>> Paul Spencer >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> Paul Spencer >>>>>>>> >>>>>>> >>>>> >> > ------=_Part_9510_31853450.1213374212739--