jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shmuel Krakower <shmul...@gmail.com>
Subject Re: Is my load server causing results to be in-accurate?
Date Thu, 10 Oct 2013 05:24:14 GMT
Hi Adrian,
I agree with all of you just wrote.
Nowadays I don't want to spend that time to pushing the limits of the
injector/load generator and finding out when it is starting to affect the
results. Sometimes this trial and error effort is just too expensive in
compare to the total project budget or timelines.

Instead I wanna be on the safe side from day one. Big servers are cheap and
easy to get today. But that may not be true for everyone.

Regarding the point of using another monitoring during the time of the load
test, that's always a good practice. I'm glad that you've raised that
point. I invite anyone who's load testing a system available from the
internet to use my monitoring service which run JMeter scripts(you simply
drop your load test script and use it for monitoring). It will give you the
insight of how the SUT respond to other clients around the globe, while you
run your test from specific injector/location.

It is www.beatsoo.org
 On Oct 9, 2013 8:03 PM, "Adrian Speteanu" <asp.adieu@gmail.com> wrote:

> Hi,
>
> I disagree with the previous message. The principle is good, but our
> approaches differ even if we start from the same general idea. For short I
> would max out the test injector as much as possible. And by as much as
> possible I mean that you don't want that the way you generate load is
> affecting the final results. That's true. But in order to know that for
> sure, you should validate your assumptions on how your script will actually
> work prior to running an important test!! This is a very important idea
> that I can't stress enough.
>
> Just by not using the entire CPU, you're not in the safe zone, maybe
> something in your scenario isn't valid and impedes the test from generating
> load in a predictable fashion (and by predictable, I mean that it won't
> surprise the person that created the script, not making a monotonous
> script) or maybe it uses too much resources and so on.
>
> Also, if you use a large portion of the machine's resources, it does NOT
> imply that your test will be "unstable" and the results affected. IF the
> resource consumption and the generated throughput is predictable and
> stable, then you did a good job for one, and you can scale up the CPU
> consumption (I would leave 10% available to the system, on an EC2 instance,
> if the CPU usage doesn't spike during the test, so look for max average
> values, not an absolute average usage). In 99% of the times you should be
> fine. Especially when the test is 1,000 threads complex. When designing
> tests with more threads, the CPU's ability to cycle them might become a
> bottleneck, but again you can validate this a priori by gradually
> increasing the traffic generated, by creating a mock script that doesn't
> hit the servers and uses test client resources the same way as the actual
> script will, by monitoring the client machine, by comparing client side
> results with server side results, etc. Apart from the CPU, there's also
> some variation added by the networking involved, the monitoring needs to be
> broad enough (and it depends on what is the resource that the script will
> stress more).
>
> Does it affect results? It can, this is why you need to prepare the test
> and to be able to compare results with a third party tool. So the second
> part of making sure the test is ok is having a monitoring system in place
> on the server side as well. All production servers use some level of
> monitoring, so why not have it up and running by the time you test? Its a
> must imho.
>
> Once your test performs as you intend it, then you won't have to worry
> about test injector machines' resource consumption, because you should know
> exactly how it will behave. If resources are in check then, the test
> results won't be affected either.  Unless it has a bug that we don't know
> about :). For simple metrics, I don't think it has.
>
> Cheers,
> Adrian S
>
>
>
> On Wed, Oct 9, 2013 at 7:54 AM, Shmuel Krakower <shmulikk@gmail.com>
> wrote:
>
> > From my experience the CPU on the load generator should remain idle if
> you
> > want to get stable results from the test, such results where you can
> > actually compare with previous tests without the question of whether the
> > load generator is overloaded.
> >
> > A gross number I go with is at the range of 30% CPU usage or less on the
> > JMeter host.
> >
> > Of-course you always need to take into consideration the memory is not
> > swapped in OS level (so it can be ~100% RAM usage).
> > One other thing is JMeter JVM GC workloads, which should be minimal, you
> > should make sure JMeter is not spending too much effort on GC, no matter
> > what's the configured heap size.
> >
> > But that's a good question, I'd also like to know what others are doing.
> >
> >
> >
> > Shmuel Krakower.
> > www.Beatsoo.org - re-use your jmeter scripts for application performance
> > monitoring from worldwide locations for free.
> >
> >
> > On Wed, Oct 9, 2013 at 5:08 AM, Tim Koopmans <tim@altentee.com> wrote:
> >
> > > At flood.io we find a better measure of performance and impact on test
> > > results is JVM heap utilization.
> > >
> > > For example, this benchmark https://flood.io/954b7d5d79f134 shows
> > > degradation of response time over time as heap utilization increases
> > >
> > >
> >
> https://github.com/flood-io/flood-loadtest/blob/master/benchmarks/results/954b7d5d79f134.md
> > >
> > > Having said that we were running 30K users on a single JVM. You can
> find
> > > out more about our benchmarks here:
> > > https://flood.io/blog/11-benchmarking-jmeter-and-gatling
> > >
> > > You can correlate increased CPU of course with heavy resource
> utilization
> > > within the JVM, but looking at CPU alone is like trying to measure
> > rainfall
> > > by listening to it fall on the roof.
> > >
> > > Regards,
> > > Tim
> > >
> > >
> > >
> > >
> > > Tim Koopmans
> > > +61 417 262 008
> > >
> > > <http://altentee.com/>
> > >
> > > The Automation Company
> > >
> > >
> > >
> > > On Wed, Oct 9, 2013 at 1:00 PM, Ophir.Prusak <
> > ophir.prusak@blazemeter.com
> > > >wrote:
> > >
> > > > I'm running a JMeter test using JMeter on an amazon EC2 instance
> > (large)
> > > as
> > > > the load server using 1,000 threads. The load server CPU is steady at
> > > about
> > > > 90% utilization and memory is at 70%.
> > > >
> > > > Is there a rule of thumb regarding at what point does the server not
> > > having
> > > > enough resources impact test results?
> > > >
> > > > Regarding CPU would you say 90%? 95% 99%? Regarding Memory would you
> > say
> > > > 90%? 95% 99%?
> > > >
> > > > Thanks Ophir
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://jmeter.512774.n5.nabble.com/Is-my-load-server-causing-results-to-be-in-accurate-tp5718385.html
> > > > Sent from the JMeter - User mailing list archive at Nabble.com.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > > For additional commands, e-mail: user-help@jmeter.apache.org
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message