Mailing-List: contact jmeter-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "JMeter Users List" <jmeter-user@jakarta.apache.org>
Received-SPF: softfail (athena.apache.org: transitioning domain of
 oliver_lloyd@hotmail.com does not designate 216.139.236.26 as permitted
 sender)
Date: Sun, 29 May 2011 11:01:01 -0700 (PDT)
From: oliver <oliver_lloyd@hotmail.com>
To: jmeter-user@jakarta.apache.org
Message-ID: <1306692060999-4437493.post@n5.nabble.com>
In-Reply-To: <BANLkTikTMVx7ZY=tyX7duOgeCUrTCYaNKg@mail.gmail.com>
References: <BANLkTikRZ04RmnGoK420TCkoxPkg1ccunQ@mail.gmail.com>
 <BANLkTikTMVx7ZY=tyX7duOgeCUrTCYaNKg@mail.gmail.com>
Subject: Re: How are throughput and response time related?
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

"Throughput is inversely proportional to Response Time." Er, that's not true.

It is worrying how often people fail to grasp this basic concept. It's
really not rocket science guys.

I'm feeling generous so let's break it down. 'Throughput' is actually formed
of two words, 'through' and 'put', as in how much can I 'put' 'though'. It
is a *rate* so it is measured in an amount over time. Typically, in
computing it is a rate of bytes per second. But you can also think about
gallons of water flowing through a pipe if this helps.

In every system, electronic or otherwise, there is a limit - a maximum
throughput of some sort. Once this is reached then you would expect the
system to start behaving badly, this usually results in longer response
times but not necessarily. You might also see errors, HTTP503s and the like.
It depends.

Look, if my test is making 10 requests a second and each of those requests
is 10kb in size then my current throughput is 100kb/ps. At the same time I
might see avg. response times of 300ms.

If I double the load to 20 requests a second then my throughput increases to
200kb/ps. What happens to my avg. response time? Will it automatically
increase to 600ms? Of course not! It might, sure, who knows right? This is
why we test stuff. But it could equally just stay the same at 300ms.

But at some point, if you keep on increasing throughput you will eventually
reach a limit. Say I ramp up the test to 100 reqs/per sec and at this point
I start to see response times grow. Now, typically, even if I try to
increase the rate or requests I will not be able to make the throughput go
any higher - this is called a bottleneck. All that will happen if I increase
the load is I will get longer and longer response times until it all gets a
bit messy. In this scenario, my maximum throughput was shown to be
1000kb/ps. 

But the affect of reaching a bottleneck could be anything, not JUST greater
response times. You might continue to see response times of 300ms but start
to get failures. Or, maybe a back end process starts to slow down or fail,
or memory may start to grow, or images no longer get rendered properly, or
blah, blah, blah. If you only focus on response times you are possibly
(probably) not testing your system properly. 

But what if I change the test I ran earlier to make a call to a different
page? This time the page is bigger with a size of 20kbs - twice the size of
the old one. Now, I rerun the test again and I again  reach 100 reqs/per sec
and then I hit a bottleneck. But this time the maximum throughput I got to
was 2000kb/ps! But hang on, I thought the maximum throughput was 1000kb/ps!
Not neccessarily, what if the bottleneck was actually not bandwidth but
somewhere else, like say, in the database. 

You are surely going to fail if you try to analyse your system by only
looking at throughput in bytes per sec vs. response times. You could run the
first test and proclaim to the world that you've found the maximum
throughput! You would run around the office shouting "this system can only
reach 1000kb/ps! Let's buy a bigger Switch box, I want 100Gbit ethernet
cards! I demand fiber optics everywhere!" And then your company might spend
10 gadzillion dollars installing all this crap and you rerun your test and
still get the same 1000kb/ps bottleneck. You see how you might look a bit
stupid at this point?

This is why it is eternally better to talk about throughput in business
terms. Stop poring over at those sexy bytes per second graphs, go on, stop
it, it'll make you go blind staring at that stuff. Instead, start looking at
some real data that tells you the rate of requests for your *business*
transactions. Like: logins per second, registrations per second, payments
per second. It's unlikely that you will actually reach a bandwidth limit
(I've had this twice in 15 years vs. 100s of other types of bottlenecks) so
what you should really care about is throughput in terms of business
transactions per second (hour, minute, whatever). Or API calls a second, or
page views, or whatever works for you best.

I'm bored now but if you don't get these simple concepts then you should not
be trying to run performance tests. My advice to anyone who feels
enlightened by this text is to stop what you are doing, take your hands away
from JMeter, you are not helping anyone. Instead, open a browser and spend a
few hours reading wikipedia and friends. I promise you, if you teach
yourself these basics you will find that you start to enjoy your job more,
your sense of self worth will increase and you will become a happier person.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437493.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org