Mailing-List: contact users-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@httpd.apache.org
Received-SPF: pass (nike.apache.org: domain of random.danno@gmail.com
 designates 209.85.223.177 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=mwhqS9mSnTWtfeb9D21jiE7+WMLPO3ls39c7gdchbOMz65AI6d9W0Yf9FSEM44H2w9
         BLm7Pfv8nBDfA+4m2jgC2v9KDYKguUQ+1M2IjDqLoT3CHQbZ3W1NTjeMEkp3QtbyPExp
         fSdA5VENuwNdTR71mn/Bdhyxe2BPCUaBYvV8c=
MIME-Version: 1.0
In-Reply-To: <6925cd581002171429v7b36138byf434da7e5d8a9433@mail.gmail.com>
References: <6925cd581002151037y518469e1ha94e82fc0faf1c64@mail.gmail.com>
	 <0AF9ABB5-2848-47D4-962B-CA2F08F1FA59@berkeley.edu>
	 <6925cd581002171206t54173a14ne59357e1c2b17c57@mail.gmail.com>
	 <6B6FDAC6-05E6-431D-8863-2102CC763170@berkeley.edu>
	 <6925cd581002171250q97d99dbr59f71c615a8cd9cb@mail.gmail.com>
	 <5FAA0173-F4FD-4CBE-86F1-93A6DD8BF846@berkeley.edu>
	 <6925cd581002171429v7b36138byf434da7e5d8a9433@mail.gmail.com>
Date: Fri, 19 Feb 2010 13:24:10 -0600
Message-ID: <6925cd581002191124y48108f29h20bf390b288972b2@mail.gmail.com>
From: Dan Denton <random.danno@gmail.com>
To: users@httpd.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [users@httpd] JMeter Load Testing of Tomcat through Apache Proxy

On Wed, Feb 17, 2010 at 4:29 PM, Dan Denton <random.danno@gmail.com> wrote:
> On Wed, Feb 17, 2010 at 3:19 PM, Robert Hall <rfhall@berkeley.edu> wrote:
>>
>> On Feb 17, 2010, at 12:50 PM, Dan Denton wrote:
>>
>>> On Wed, Feb 17, 2010 at 2:26 PM, Robert Hall <rfhall@berkeley.edu> wrot=
e:
>>>>
>>>> On Feb 17, 2010, at 12:06 PM, Dan Denton wrote:
>>>>
>>>>> On Mon, Feb 15, 2010 at 12:53 PM, Robert Hall <rfhall@berkeley.edu>
>>>>> wrote:
>>>>>>
>>>>>> Dan,
>>>>>>
>>>>>> On Feb 15, 2010, at 10:37 AM, Dan Denton wrote:
>>>>>>
>>>>>>> Hello all. I=E2=80=99m trying to load test a login page served by t=
omcat 6,
>>>>>>> proxied through apache 2 with mod_proxy. I=E2=80=99m using JMeter 2=
.3.4 to
>>>>>>> conduct the testing. My thread group consists of 500 sessions , and
>>>>>>> the sample is a GET of a simple login page.
>>>>>>>
>>>>>>> JMeter returns errors for a varying percentage of the samples. The
>>>>>>> errors returned are generally the following:
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTP=
SamplerBase.java:1037)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTP=
SamplerBase.java:1023)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread=
.java:346)
>>>>>>> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:243=
)
>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>
>>>>>>> The issues I=E2=80=99m having are twofold. I=E2=80=99m having diffi=
culty determining
>>>>>>> if these errors are coming from JMeter or Tomcat, as they=E2=80=99r=
e displayed
>>>>>>> in the response window of JMeter. The developers think the error is
>>>>>>> coming from JMeter given the last few lines of the trace above.
>>>>>>
>>>>>> The developers are correct.
>>>>>>
>>>>>>> Given that I'm not a programmer I should probably take their word f=
or
>>>>>>> it,
>>>>>>> but why would JMeter show this error as the response?
>>>>>>
>>>>>> The system you are running JMeter on isn't able to handle the load.
>>>>>>
>>>>>>> Second, I've tried tweaking my process counts (startservers, maxspa=
re,
>>>>>>> etc...) with no change in the outcome. I can mitigate the issue by
>>>>>>> pointing JMeter directly to tomcat, but I need this product to go
>>>>>>> through our apache proxy for SSL.
>>>>>>>
>>>>>>> Any help on this would be greatly appreciated.
>>>>>>
>>>>>> There must be some JMeter setting that will work otherwise you would=
 be
>>>>>> unable
>>>>>> to access the webapp over SSL from the system that is hosting JMeter=
.
>>>>>>
>>>>>> Try reducing everything to a count of 1 in JMeter.
>>>>>>
>>>>>> If that doesn't work, there is a problem with the SSL config in JMet=
er;
>>>>>> google "jmeter ssl".
>>>>>>
>>>>>> Otherwise, try spreading the load our across several JMeter instance=
s
>>>>>> installed on separate systems.
>>>>>>
>>>>>> - Robert
>>>>>>
>>>>>
>>>>> Thanks for the reply Robert. I've set up JMeter 4 slaves, each with a=
t
>>>>> least two 2.8 Ghz procs and 2 GB of RAM, and still regardless of
>>>>> whether it's 1 node simulating 400 sessions or 4 nodes each simulatin=
g
>>>>> 100, I still see these errors at 400 sessions or more. Also, when I
>>>>> use multiple slaves to execute the test, the percentage of failures
>>>>> when simulating 400 sessions is greater and the failures happen
>>>>> earlier in the test.
>>>>>
>>>>> This makes me think that this isn't just an issue with the systems
>>>>> running JMeter, but I'm not sure. I've tried tweaking my SSL Session
>>>>> Timeout as well, but with no effect. I did this because watching the
>>>>> mod_status page on this apache instance, I can see the current sessio=
n
>>>>> count top out at about 330 every time, then subside. My guess was tha=
t
>>>>> SSL sessions were somehow bottlenecking.
>>>>>
>>>>> If anyone has any other suggestions, they would be greatly appreciate=
d.
>>>>>
>>>>
>>>> Dan, this sounds like an Apache httpd configuration issue.
>>>>
>>>> Check the values for the 'KeepAliveTimeout' and 'KeepAlive' directives=
,
>>>> http://httpd.apache.org/docs/2.0/mod/core.html#keepalive
>>>>
>>>> I suggest using "KeepAlive =C2=A0On" and "KeepAliveTimeout 1"; the lat=
ter is
>>>> probably defaulted to 15.
>>>>
>>>> - Robert
>>>>
>>> Thanks again Robert. My apache proxy has both directives already set,
>>> and KeepAliveTimeout was indeed set to 15. I tried running the 400
>>> session test across 4 slaves with KeepAliveTimout set to 1 and 30, and
>>> the same result. A failure rate of approximately 20 percent.
>>>
>>> Are there any other directives you suggest changing?
>>>
>>> Thanks...
>>>
>>
>> Surprising that KeepAliveTimeout of 1 and 30 produced the same results.
>>
>> Other directives: (with KeepAliveTimeout of 1)
>>
>> MaxKeepAliveRequests - defaults to 100, try raising to 500
>>
>> Separately, set KeepAlive Off.
>>
>> Have you looked at the httpd error logs?
>>
>> - Robert
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Projec=
t.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> =C2=A0" =C2=A0 from the digest: users-digest-unsubscribe@httpd.apache.or=
g
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> Robert,
>
> With KeepAliveTimout set to 1, and with MaxKeepAliveRequests set to
> either 100 or 500, the same results. When using 4 nodes, 100 sessions
> a piece, failure rates are between 10 and 30%. With individual nodes
> firing off 400 sessions, failure rates are 10% or less.
>
> With KeepAlive off, the same results. The results are somewhat random,
> but around the same failure rates.
>
> Nothing in the error logs that seems to point to this issue. Only
> occasional messages regarding child processes during the restarts for
> the configuration changes.
>
> So far, none of these changes seem to have had an effect. The failure
> rates are the same regardless.
>
> Thanks for the help...
>

Robert,

Thanks for the reply. There are no firewalls in play here. The
original system being tested is a development system (not as robust as
our production systems), and the proxy and the tomcat instance serving
the webapp were on the same server. I moved the proxy to another
machine and that helped push the count to around 500 before we see an
error rate around 3%. Based on this, it's very likely this is a case
of simple resource limitations on the server.

I will be doing more testing next week with hardware more similar to
our production systems, which have about 400% more horsepower than our
dev systems.

Thanks again for your help!

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org