jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <seb...@gmail.com>
Subject Re: Design of Timeout test element / sampler interrupter
Date Tue, 01 Sep 2015 23:44:14 GMT
On 1 September 2015 at 23:48, Milamber <milamber@apache.org> wrote:
>
>>> * On basic scenario, with 2 http requests on the same url, Callable
>>> option
>>> seems most quick compare to Runnable (~70% more quick)
>>
>> I'm surprised it makes that much difference.
>>
>> What were you measuring?
>
>
> The URL below (in redhat documentation website)
>
>> How many threads?
>
>
> One VU with 10 loops
>
> The latest version of the branch seems have some differences:
>

OK, I see. Thanks again.

> 2015/09/01 23:41:14 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 3.15911E-4
> secs
> 2015/09/01 23:41:16 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 7.38335E-4
> secs
> 2015/09/01 23:41:18 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 7.26091E-4
> secs
> 2015/09/01 23:41:20 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 7.4593E-4
> secs
> 2015/09/01 23:41:22 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 0.00181855
> secs
> 2015/09/01 23:41:24 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 5.33232E-4
> secs
> 2015/09/01 23:41:26 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 4.05846E-4
> secs
> 2015/09/01 23:41:28 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 5.50154E-4
> secs
> 2015/09/01 23:41:30 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 6.93174E-4
> secs
> 2015/09/01 23:41:32 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 6.11455E-4
> secs
> 2015/09/01 23:41:34 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 6.1518E-4
> secs
> 2015/09/01 23:41:36 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 6.24097E-4
> secs
> 2015/09/01 23:41:38 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 7.53767E-4
> secs
> 2015/09/01 23:41:40 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 6.84471E-4
> secs
> 2015/09/01 23:41:42 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 7.34266E-4
> secs
> 2015/09/01 23:41:44 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 5.88123E-4
> secs
> 2015/09/01 23:41:46 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 3.34427E-4
> secs
> 2015/09/01 23:41:48 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 6.3848E-4
> secs
> 2015/09/01 23:41:50 WARN  - jmeter.timers.InterruptTimer: Run Done
> interrupting HTTPSamplerProxy @469583928 'HTTP Request 1'  took 7.77866E-4
> secs
> 2015/09/01 23:41:52 WARN  - jmeter.timers.InterruptTimer: Call Done
> interrupting HTTPSamplerProxy @508309015 'HTTP Request 2'  took 7.13642E-4
> secs
>
> The record times is very close (in nano seconds), perhaps, not important
> (significant).

The Run and Call versions use the same code to interrupt the sampler;
the only difference is how the interrupt task is scheduled.
So this should not affect the amount of time it takes to perform the interrupt.
The recorded times don't seem to show a significant difference - apart
from one outlier value they are all around 300-800 microseconds (I
think I've got the scaling right)

However the scheduling method may affect the overall performance of the test.
Starting a Runnable is likely to be more expensive than invoking a Callable.

Whether this difference is significant or not remains to be seen.
But given that the interrupt does not take a significant amount of
time, it may not be worth worrying about.
The code can use Callable by default.

I've done some work on implementing the test element as a PreProcessor
but this is not quite finished.

>
>
>
>>
>>> * Basic tests with the same url + Assertion Duration : the behavior is ok
>>>
>>> * Basic tests with the same url + a Constant Timer set to 2 sec (and
>>> Interrupt Timer set to 2 sec), the results seems not ok: the url is fully
>>> retrieve with a response time > 5 seconds (the request isn't interrupt,
>>> the
>>> response tab show the HTML code of the page). If I disable the Constant
>>> Timer, the request is interrupt in 2 sec. (the response tab show a
>>> (normal)
>>> exception causing by the interruption)
>>> Probably an issue...
>>
>> No, this is expected because the InterruptTimer is called before the
>> delay is performed.
>>
>> I have not yet implemented the use of sampleStarted/sampleStopped
>> which will allow the test element to be called after any other timers.
>>
>> At present the interrupt test element has no way to allow for other
>> timer delays because it does not know what they will be.
>> These occur after it is called (and before the sample).
>> That is why sampleStarted/sampleStopped are needed.
>> Or some other change is needed to JMeterThread that will allow the
>> interrupt timer to schedule the interrupt at the correct time.
>>
>>> * The fields Task type and Call delay isn't localized and the values
>>> aren't
>>> reset with your add a second Interrupt timer (I suppose you knows that,
>>> current branch is a PoC)
>>
>> Yes, they are for testing purposes only and aren't needed in the final
>> release.
>>
>>> Milamber
>>>
>>> [the URL for test (deflate is disable in HTTP Header):
>>>
>>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.2/html-single/Administration_and_Configuration_Guide/index.html
>>> ]
>>>
>>>
>>>
>>>
>>> On 30/08/2015 17:37, sebb wrote:
>>>>
>>>> OK, the SampleTimeout branch now has a version that use Runnable by
>>>> default, but you can use Callable (type==CALL) with an optional delay
>>>> to simulate an interrupt that takes a long time.
>>>>
>>>> It works fine for me in simple tests.
>>>>
>>>> If you can break it, so much the better.
>>>>
>>>>
>>>> On 30 August 2015 at 16:36, sebb <sebbaz@gmail.com> wrote:
>>>>>
>>>>> On 30 August 2015 at 11:54, sebb <sebbaz@gmail.com> wrote:
>>>>>>
>>>>>> On 30 August 2015 at 07:12, Philippe Mouawad
>>>>>> <philippe.mouawad@gmail.com> wrote:
>>>>>>>
>>>>>>> On Sunday, August 30, 2015, sebb <sebbaz@gmail.com> wrote:
>>>>>>>
>>>>>>>> I've had a look at the classes that implement SampleListener,
and
>>>>>>>> apart from ResultAction and TransactionSampler, only the
Listeners
>>>>>>>> use
>>>>>>>> it. Since usage of these should be minimised in a production
test,
>>>>>>>> it's likely that there won't be as many implementations as
I had
>>>>>>>> feard.
>>>>>>>
>>>>>>> Implementation of SampleListener ?
>>>>>>> Usage of which should be minimised ?  SampleListener ?
>>>>>>
>>>>>> Usage of additional  Listeners should be minimised in a production
>>>>>> test.
>>>>>>
>>>>>>>> Also if the implementation is empty, the overhead will be
quite
>>>>>>>> small.
>>>>>>>>
>>>>>>>> [There is a work-round if it does prove expensive: the
>>>>>>>> SampleListener
>>>>>>>> interface could be split into two parent interfaces.]
>>>>>>>>
>>>>>>>> So assuming that JMeterThread implements
>>>>>>>> sampleStarted/sampleStopped,
>>>>>>>> the Timeout element can use the Start to set up the timer
and the
>>>>>>>> Stop
>>>>>>>> to cancel it. This will reduce the number outstanding as
much as
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> The timeouts have to be implemented using separate threads
for two
>>>>>>>> reasons:
>>>>>>>> - it's obviously not possible to interrupt a sampler from
the same
>>>>>>>> thread as the sampler
>>>>>>>> - depending on the sampler, and its state, the interrupt
may take a
>>>>>>>> while to complete, so each interrupt must be done in its
own thread
>>>>>>>
>>>>>>> Are you sure, calling interrupt is usually just about setting
a flag
>>>>>>> no?
>>>>>>
>>>>>> Ultimately yes, I guess a flag will be set.
>>>>>> However I'm not sure that it is always instantaneous, as there may
be
>>>>>> locks to aquire.
>>>>>>
>>>>>>> Having 1 thread for each interruption, could lead to hundred
of
>>>>>>> threads running for high throughput threads (500 res/s for example),
>>>>>>> it
>>>>>>> won't scale.
>>>>>>
>>>>>> That assumes that all the samples in all the threads have timeouts
>>>>>> enabled.
>>>>>
>>>>> It also assumes that the threads are running for the life of the
>>>>> timeout, which I've just realised is not the case.
>>>>>>>
>>>>>>> Why can't we have 1 Thread (TimeoutChecker) called every N
>>>>>>> milliseconds
>>>>>>> that checks all registered JMeterThreads to check and call interrupt
>>>>>>> if
>>>>>>> necessary ?
>>>>>>>
>>>>>> That's quite a bit of work to code, but if there is already such
a
>>>>>> queing mechanism it would be worth trying.
>>>>>
>>>>> Actually the ScheduledExecutorService does just that.
>>>>> It uses one thread to handle the timeouts in sequence, and then starts
>>>>> a new Runnable when the timeout expires.
>>>>>
>>>>> Now that I have created the service as a singleton, there will only be
>>>>> one extra thread (the executor) most of the time.
>>>>> Since the timer tasks are cancelled if the sample completes in time,
>>>>> there will only ever be at most one extra thread for each overdue
>>>>> sample.
>>>>>
>>>>> I think it is a reasonable assumption to assume that the number of
>>>>> such samples will generally be small.
>>>>> At the very most it could only be one per thread.
>>>>> And remember that the new thread is only started when the timeout
>>>>> expires.
>>>>> It is extremely unlikely that they will all expire at once, and anyway
>>>>> (according to you) they don't take long to run so they won't build up.
>>>>>
>>>>> I've just noticed that there is a version of the Executor Service that
>>>>> uses Callable rather than Runnable (not sure why I missed it before).
>>>>> I asume this means the tasks are run in the same thread (I'll check
>>>>> this).
>>>>> If the interrupts really do happen quickly, this might be a better
>>>>> choice.
>>>>> Even if not, then it may not matter so long as any delays don't
>>>>> continue to build up.
>>>>>
>>>>>>>> It should be possible to use a single shared instance of
the
>>>>>>>> ScheduledExecutorService; that could be lazily created using
IODH.
>>>>>>>> [I
>>>>>>>> can try that with the current implementation]
>>>>>>
>>>>>> I've already tried it.
>>>>>>
>>>>>>> See my note above
>>>>>>>>
>>>>>>>> As to whether the Timeout class should be a Timer or some
other type
>>>>>>>> of test element - that does not matter so long as it can
be applied
>>>>>>>> to
>>>>>>>> the samplers individually or when in scope.
>>>>>>>>
>>>>>>>> I chose Timer because it was already called in the right
place, but
>>>>>>>> I
>>>>>>>> assume JMeterThread can call any Test class provided that
it
>>>>>>>> implemented the SampleInterface.
>>>>>>>>
>>>>>>>> It must be one of the existing Test Element classes that
are handled
>>>>>>>> by the Menu system otherwise it will need special handling.
>>>>>>>>
>>>>>>>> The scope requirement rules out Config elements and Logic
>>>>>>>> Controllers.
>>>>>>>
>>>>>>> It does not seem like a pre-processor to me, nor a post-procesor,
nor
>>>>>>> a
>>>>>>>>
>>>>>>>> Listener
>>>>>>>>
>>>>>>>> So AFAICT the only remaining options are the Timer and Assertions.
>>>>>>>>
>>>>>>>> I think both are justifiable.
>>>>>>>
>>>>>>> Why isn't it part of Sampler abstract class and as such a field
in
>>>>>>> Sampler?
>>>>>>
>>>>>> How does the user indicate that a Timeout should be applied to a
>>>>>> particular Sampler?
>>>>>> It's a lot of work to add Timeout fields to every GUI.
>>>>>> Whereas being able to add a child test element to each applicable
>>>>>> sampler is already supported.
>>>>>> Further, such a test element can be applied to multiple samplers
in
>>>>>> scope.
>>>>>> Much easier to enable and disable a single element that having to
>>>>>> update each Sampler.
>>>>>>
>>>>>> That's why it needs to be a separate test element.
>>>>>>
>>>>>>> For me none of Timer nor Assertions are conceptually valid.
>>>>>>> The behaviour is not a pause(so not Timer for me), it's not an
>>>>>>> Assertion
>>>>>>> neither as for me an Assertion only checks something.
>>>>>>> Although not fully satisfying it look to me more of a PreProcessor
as
>>>>>>> it
>>>>>>> sets a timeout on the Sampler , it can also be considered as
a Post
>>>>>>> Processor.
>>>>>>
>>>>>> OK, I could live with it being a Pre-Procesor.
>>>>>> That has the correct scoping rules.
>>>>>>
>>>>>>>> The name of the class can of course be changed from InterruptTimer
-
>>>>>>>> I
>>>>>>>> think that is probably not the best choice. Maybe something
like
>>>>>>>> SamplerTimeout?
>>>>>>>>
>>>>>>> Yes or SamplerTimeouter or SamplerInterrupter
>>>>>>>
>>>>>> Or even SampleTimeout - that's what it does, it applies a timeout
to a
>>>>>> sample.
>>>>>>
>>>>>>> --
>>>>>>> Cordialement.
>>>>>>> Philippe Mouawad.
>>>
>>>
>

Mime
View raw message