Return-Path: 
 <user-return-11213-apmail-geronimo-user-archive=geronimo.apache.org@geronimo.apache.org>
Delivered-To: apmail-geronimo-user-archive@www.apache.org
Received: (qmail 47119 invoked from network); 13 Aug 2008 00:32:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Aug 2008 00:32:59 -0000
Received: (qmail 12123 invoked by uid 500); 13 Aug 2008 00:32:57 -0000
Delivered-To: apmail-geronimo-user-archive@geronimo.apache.org
Received: (qmail 12103 invoked by uid 500); 13 Aug 2008 00:32:57 -0000
Mailing-List: contact user-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
list-help: <mailto:user-help@geronimo.apache.org>
list-unsubscribe: <mailto:user-unsubscribe@geronimo.apache.org>
List-Post: <mailto:user@geronimo.apache.org>
Reply-To: user@geronimo.apache.org
List-Id: <user.geronimo.apache.org>
Delivered-To: mailing list user@geronimo.apache.org
Received: (qmail 12092 invoked by uid 99); 13 Aug 2008 00:32:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Aug 2008 17:32:57 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kevan.miller@gmail.com
 designates 64.233.184.236 as permitted sender)
Received: from [64.233.184.236] (HELO wr-out-0506.google.com) (64.233.184.236)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2008 00:31:59 +0000
Received: by wr-out-0506.google.com with SMTP id c8so1138277wra.27
        for <user@geronimo.apache.org>; Tue, 12 Aug 2008 17:32:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:mime-version
         :subject:date:references:x-mailer;
        bh=cv6BBJoVnxcG4tLDCnlQQkhpQch8oAPdFScCpXZvu7o=;
        b=VwjcRyJ3Mw1uE15ZgTLkLjjhkYEKQ7iP0+ulZ3Cz5cxX9GI1UZ/gwjmagWWfhFM3ka
         2DtGdBgNddDz0w0aUuZNJvmK1ZddF+Xs9LID+2qvfwU/pibtVzB9u6DIDpssDNJDo8Ub
         VGXz71lqM+Pv7euMSi0Qpj+eznf2kTQ1WOchg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=TxORhj1Z7XnSRxCB0fu1e85fyFDjEob6cbDPBkFj0yEkhX29D+PC9Y+zWUVTP4/j/x
         AmMEPjYLRZS5D9guRGMBYdBPJ3QoQ2oVmo9rS/UNqsNgFWLa06Qc0FJyPrLevbKyPb3o
         x/zelrF3jHz8I7/XVAwKL8DsuMDyVaOkBT1XU=
Received: by 10.90.71.16 with SMTP id t16mr13812566aga.27.1218587529951;
        Tue, 12 Aug 2008 17:32:09 -0700 (PDT)
Received: from coltrane-009027164142.raleigh.ibm.com ( [129.33.49.251])
        by mx.google.com with ESMTPS id y56sm7230201hsb.0.2008.08.12.17.32.08
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 12 Aug 2008 17:32:09 -0700 (PDT)
Message-Id: <5BB42E86-180C-439C-A881-EC1DAAA94EE1@gmail.com>
From: Kevan Miller <kevan.miller@gmail.com>
To: user@geronimo.apache.org
In-Reply-To: <18948990.post@talk.nabble.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: large-scale app.. random spiking on even static files!
Date: Tue, 12 Aug 2008 20:32:08 -0400
References: <18902003.post@talk.nabble.com> <18902031.post@talk.nabble.com>
 <c78909240808082109p51018d88g40e9275d05aa012f@mail.gmail.com>
 <181D9AEB-C89F-4EE5-9E77-A1A0701FE5A6@gmail.com>
 <18948990.post@talk.nabble.com>
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 12, 2008, at 2:21 PM, pc3 wrote:

>
> How does this look in terms of my GC settings?
>
> Note the Full GC taking 7-9 sec... and yes, our users are feeling it.
>
> 9294.839: [Full GC [PSYoungGen: 5669K->0K(518848K)] [PSOldGen:
> 2270713K->2036966K(3495296K)] 2276382K->2036966K(4014144K) [PSPermGen:
> 132633K->132553K(262144K)], 9.6645590 secs]

<snip>

> Kevan Miller wrote:
>>
>>
>> On Aug 9, 2008, at 12:09 AM, Pete Clark wrote:
>>
>>> Finally, here are our startup opts as it pertains to GC if it helps:
>>>
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:PermSize=256m - 
>>> XX:MaxPermSize=256m
>>> -Xmx5120m -Xms5120m -Xss128k -XX:ParallelGCThreads=20
>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8
>>> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31
>>> -XX:+AggressiveOpts -Xloggc:/var/tmp/gc.log

<snip>

What's the behavior like, if you run without the GC tuning parameters?  
Wondering how much these tuning parms are actually buying you... E.g:

-XX:+HeapDumpOnOutOfMemoryError -XX:PermSize=256m -XX:MaxPermSize=256m  
-Xmx5120m -Xms5120m -Xss128k -Xloggc:/var/tmp/gc.log

Do you have any very large objects? How much physical memory do your  
servers have?

I doubt that 20 parallel GC threads are necessary, but also doubt that  
they are causing any harm...

As I mentioned, I'm not really up-to-speed with current (1.5/1.6) GC  
tuning parameters/behaviors. One thing to explore is how to keep  
objects out of your tenured space (and in young space). Hmm. Have you  
tried -XX:+UseParallelOldGC?

I assume your problem is in the GC of your tenured space (but that's  
kind of a big assumption... ;-)

--kevan