jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tuukka Mustonen <tuukka.musto...@gmail.com>
Subject Re: JMeter 3.1 httpclient4.retrycount does not work
Date Wed, 15 Mar 2017 15:45:13 GMT
Ok, I added test that uses POST/PATCH/DELETE/GET methods.

With #6020 build of JMeter
and httpclient4.request_sent_retry_enabled=false, there were some retries,
but mostly not. I think it retried GET (and maybe DELETE) methods but not
anything else.

With httpclient4.request_sent_retry_enabled=true, all failed requests were
retried and this resulted in zero errors.

So yeah, this new option is something I would need/want with ALB (until
they fix the issue, I still haven't received reply if AWS considers it a
bug or a feature).

Though I am still not sure what it actually does:

*"However, can you crystalize what that new property is supposed to do? As
I wrote above I thought it was to retry also non-idempotent methods, but I
guess it's actually supposed to do something else?"*

Tuukka


On Fri, Mar 10, 2017 at 12:08 PM, Tuukka Mustonen <tuukka.mustonen@gmail.com
> wrote:

> > You confirm it's this night nightly or yesterday's one ?
> > Did you use the property:
> > httpclient4.request_sent_retry_enabled=true
>
> It was this build: https://builds.apache.org/job/JMeter-trunk/6020/
>
> No, I didn't enable that property - I thought that property was to also
> retry non-idempotent methods.
>
> > If you used httpclient4.request_sent_retry_enabled=true and it works
> while
> > it doesn't with httpclient4.request_sent_retry_enabled=false
> > It's a good reason to add it.
> > BUt if it already works with :
> > httpclient4.request_sent_retry_enabled=false
> >
> > Then it might not be useful. Can you give feedback on this ?
>
> Yeah, I didn't declare it at so I guess it's not needed then...
>
> However, can you crystalize what that new property is supposed to do? As I
> wrote above I thought it was to retry also non-idempotent methods, but I
> guess it's actually supposed to do something else?
>
> > And we'll wait a week for your test on non idempotent methods.
>
> Ok.
>
> > Yes but it's not a good setting, in this case better use stalecheck.
>
> Ok.
>
> Tuukka
>
>
> On Fri, Mar 10, 2017 at 11:27 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>> On Fri, Mar 10, 2017 at 8:08 AM, Tuukka Mustonen <
>> tuukka.mustonen@gmail.com>
>> wrote:
>>
>> > > > +DELETE also like sebb said.
>> > > >
>> > > True but iIt was not concerned by bug.
>> >
>> > Maybe I got something wrong, but you did paste DELETE originally as
>> > non-idemponent in list of non-retryable methods:
>> >
>> > Delete not being implementation of HttpEntityEnclosingRequest, it was
>> not
>> concerned by issue.
>> Yes I thought wrongly Delete was not idempotent
>>
>> > > - Idempotent HTTP methods which are by default those that do not
>> > implement
>> > > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, PATCH, GET With
>> > body
>> >
>> > > 1/ To diagnose set those classes in debug mode:
>> > > org.apache.http.impl.client.DefaultRequestDirector
>> > >
>> > > You should see:
>> > > Retrying connect to
>> > > Retrying request to
>> > >
>> > > Log level can be set in log4j2.xml
>> >
>> > Ok, with latest nightly build, retrying works just fine, I got:
>> >
>> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: I/O exception
>> > (org.apache.http.NoHttpResponseException) caught when processing
>> request
>> > to
>> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80: The target
>> server
>> > failed to respond
>> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: Retrying
>> request to
>> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80
>> >
>> > However, with exact same testplan/configuration/scenario, retrying just
>> > does not work with HC4 on JMeter 3.1. Unfortunately, I wasn't able to
>> > configure logging on JMeter 3.1 so that it would print similar lines. I
>> > tried adding to user.properties:
>> >
>> > log_level.org.apache.http.impl.client.DefaultRequestDirector=DEBUG
>> >
>> > But that didn't do anything. Also tried tuning bin/log4j.conf. Maybe
>> that
>> > class does not exist in JMeter 3.1?
>> >
>> > Anyway, I can confirm that retying with HC4 does not work on JMeter 3.1
>> but
>> > does work on latest nightly.
>>
>>
>> You confirm it's this night nightly or yesterday's one ?
>> Did you use the property:
>> httpclient4.request_sent_retry_enabled=true
>>
>>
>> > Not for me at least (I don't believe I have
>> > anything special in my JMeter 3.1 installation, just same plugins that I
>> > downloaded into nightly build also).
>> >
>> > > I have added a property that you can set and try in user.properties:
>> > > httpclient4.request_sent_retry_enabled=true
>> > > Please give your feedback rapidly, so that we decide what to do if it
>> > has an
>> > effect.
>> >
>> > Unfortunately, I don't have a test case with non-idempotent method at
>> hand
>> > just now. I can give it a try at the start of next week.
>> >
>>
>> If you used httpclient4.request_sent_retry_enabled=true and it works
>> while
>> it doesn't with httpclient4.request_sent_retry_enabled=false
>> It's a good reason to add it.
>> BUt if it already works with :
>> httpclient4.request_sent_retry_enabled=false
>>
>> Then it might not be useful. Can you give feedback on this ?
>> And we'll wait a week for your test on non idempotent methods.
>>
>> >
>> > > But setting  httpclient4.validate_after_inactivity=0 means you
>> disable
>> > the
>> > > validation which is not a good idea as you'll end up having broken
>> > > connection.
>> >
>> > Maybe httpclient4.validate_after_inactivity=1 then?
>>
>>
>> Yes but it's not a good setting, in this case better use stalecheck.
>>
>>
>> > I assume that's pretty
>> > much the same as the old stale check?
>> >
>>
>>
>>
>> >
>> > Tuukka
>> >
>> >
>> > On Thu, Mar 9, 2017 at 10:38 PM, Philippe Mouawad <
>> > philippe.mouawad@gmail.com> wrote:
>> >
>> > > On Thu, Mar 9, 2017 at 5:28 PM, Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Sorry for the late reply,
>> > > >
>> > > > About customizing retry method whitelist:
>> > > >
>> > > > > You case is a bit soecific no ?
>> > > >
>> > > > True, it's very undesirable scenario, but that's what you get with
>> AWS
>> > > ALB
>> > > > for now - I believe there are lots and lots of other users that will
>> > face
>> > > > the same problem.
>> > > >
>> > > > I have added a property that you can set and try in user.properties:
>> > > httpclient4.request_sent_retry_enabled=true
>> > >
>> > > Please give your feedback rapidly, so that we decide what to do if it
>> has
>> > > an effect.
>> > >
>> > > Having said that, I am in discussion with AWS about if they will fix
>> it
>> > and
>> > > > with what timeline - current behavior is complete nonsense...
>> > > >
>> > > > > In this case, the bug is larger than what I thought as for now
we
>> > > > consider
>> > > > > PUT as non idempotent.
>> > > >
>> > > > +DELETE also like sebb said.
>> > > >
>> > > True but iIt was not concerned by bug.
>> > >
>> > > >
>> > > > > 1/ To diagnose set those classes in debug mode:
>> > > > > org.apache.http.impl.client.DefaultRequestDirector
>> > > > >
>> > > > > You should see:
>> > > > > Retrying connect to
>> > > > > Retrying request to
>> > > > >
>> > > > > Log level can be set in log4j2.xml
>> > > >
>> > > > Thanks! I'll try to find time to check that tomorrow...
>> > > >
>> > > > > 2/ I confirm stalecheck is still available.
>> > > >
>> > > > Can you clarify what is the difference to having
>> > > > httpclient4.validate_after_inactivity=0?
>> > > >
>> > >
>> > > It was an optimisation added by HC4 to replace stalecheck which was
>> very
>> > > costly.
>> > > But setting  httpclient4.validate_after_inactivity=0 means you
>> disable
>> > the
>> > > validation which is not a good idea as you'll end up having broken
>> > > connection.
>> > > This value should be set to a value lower than the keepalive set on
>> > server
>> > > side.
>> > >
>> > >
>> > > > Tuukka
>> > > >
>> > > >
>> > > > On Thu, Mar 9, 2017 at 2:23 PM, Philippe Mouawad <
>> > > > philippe.mouawad@gmail.com
>> > > > > wrote:
>> > > >
>> > > > > Thanks for link sebb.
>> > > > >
>> > > > > In this case, the bug is larger than what I thought as for now
we
>> > > > consider
>> > > > > PUT as non idempotent.
>> > > > >
>> > > > > I'll commit a new fix.
>> > > > > Regards
>> > > > >
>> > > > > On Thu, Mar 9, 2017 at 12:20 PM, sebb <sebbaz@gmail.com>
wrote:
>> > > > >
>> > > > > > On 9 March 2017 at 06:42, Philippe Mouawad <
>> > > philippe.mouawad@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > > > On Thursday, March 9, 2017, Tuukka Mustonen <
>> > > > tuukka.mustonen@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Hi Philippe and thanks for speedy actions!
>> > > > > > >>
>> > > > > > >> 1) Retrying only idempotent requests makes sense,
as usual.
>> But
>> > > this
>> > > > > > (retry
>> > > > > > >> on idempotent) seems to be only documented on wiki
page (
>> > > > > > >> https://wiki.apache.org/jmeter/JMeterSocketClosed).
You
>> should
>> > > add
>> > > > it
>> > > > > > also
>> > > > > > >> to http://jmeter.apache.org/userm
>> anual/component_reference.html
>> > .
>> > > > > > >>
>> > > > > > >> 2) In this case, the requests are GETs (and without
body) so
>> > they
>> > > > > > should be
>> > > > > > >> retried.
>> > > > > > >
>> > > > > > > Yes.
>> > > > > > > I debugged it and I confirm they are unless you face
the
>> > > exceptions I
>> > > > > > > mentionned.
>> > > > > > >
>> > > > > > >>
>> > > > > > >> So the bug you fixed shouldn't affect this scenario.
>> > > > > > >
>> > > > > > > Yes
>> > > > > > >
>> > > > > > >>
>> > > > > > >> There must be
>> > > > > > >> something else - any pointers to what logging module/level
I
>> > > should
>> > > > > > enable
>> > > > > > >> to inspect the (failing) retry logic?
>> > > > > > >
>> > > > > > > As you can see there is no logging in this HC4 class
>> > > > > > > Try the calling class. I' ll provide its name later
>> > > > > > >
>> > > > > > >>
>> > > > > > >> 3) AFAIK PUT is idempotent - shouldn't you retry
also that?
>> > > > > > >
>> > > > > > > No it's not IMU, it changes state of server.
>> > > > > >
>> > > > > > PUT *is* idempotent.
>> > > > > >
>> > > > > > It may change the state of the server the first time it
is sent,
>> > but
>> > > > > > subsequent PUTs should not change the state further.
>> > > > > > So it can be repeated as required.
>> > > > > >
>> > > > > > Similarly for DELETE
>> > > > > >
>> > > > > > https://tools.ietf.org/html/rfc2616#section-9.1.2
>> > > > > >
>> > > > > > >>
>> > > > > > >> 4) In this case, once I add tests for POST/PATCH/DELETE
etc.
>> I
>> > > also
>> > > > > > want to
>> > > > > > >> retry these non-idempotent requests (to tolerate
ALBs ugly
>> > > > behavior).
>> > > > > > These
>> > > > > > >> are "just" performance tests so I don't care if
I'm creating
>> > > > duplicate
>> > > > > > >> data. Is there way to specify custom method whitelist
to
>> retry
>> > any
>> > > > > HTTP
>> > > > > > >> method? I see requestSentRetryEnabled in HC4 source
code -
>> can I
>> > > > > enable
>> > > > > > >> that somehow?
>> > > > > > >
>> > > > > > > Not for now.
>> > > > > > > You case is a bit soecific no ?
>> > > > > > >
>> > > > > > >>
>> > > > > > >> 5) *Related question:
>> > > > > > >> is http.connection.stalecheck$Boolean=false (in
>> hc.parameters
>> > > file)
>> > > > > > valid
>> > > > > > >> anymore on JMeter 3.x with improved retry/stale
logic
>> > > > > > >> (and httpclient4.validate_after_inactivity)? The
line is
>> still
>> > > > there
>> > > > > in
>> > > > > > >> bundled hc.parameters file...*
>> > > > > > >
>> > > > > > > i'll double check
>> > > > > > >
>> > > > > > >>
>> > > > > > >> Tuukka
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Thu, Mar 9, 2017 at 12:25 AM, Philippe Mouawad
<
>> > > > > > >> philippe.mouawad@gmail.com <javascript:;>>
wrote:
>> > > > > > >>
>> > > > > > >> > Hello,
>> > > > > > >> > The issue with Get with body should be fixed
now:
>> > > > > > >> > - https://bz.apache.org/bugzilla/show_bug.cgi?id=60837
>> > > > > > >> >
>> > > > > > >> > Regards
>> > > > > > >> > Philippe
>> > > > > > >> >
>> > > > > > >> > On Wed, Mar 8, 2017 at 8:53 PM, Philippe Mouawad
<
>> > > > > > >> > philippe.mouawad@gmail.com <javascript:;>
>> > > > > > >> > > wrote:
>> > > > > > >> >
>> > > > > > >> > > Hello Tuukka,
>> > > > > > >> > >
>> > > > > > >> > > In my recent tests I didn't face any
issue with
>> > > > > > httpclient4.retrycount.
>> > > > > > >> > > For me it works  but be aware that JMeter
(HC4) does not
>> > retry
>> > > > all
>> > > > > > >> > > requests, it only retries those it is
allowed to :
>> > > > > > >> > > - Idempotent HTTP methods which are by
default those
>> that do
>> > > not
>> > > > > > >> > implement
>> > > > > > >> > > HttpEntityEnclosingRequest, so not POST,
PUT,DELETE,
>> PATCH,
>> > > GET
>> > > > > With
>> > > > > > >> body
>> > > > > > >> > > (<= That might be a bug thinking more
about it)
>> > > > > > >> > > - Non retriable exceptions (InterruptedIOException.class,
>> > > > > > >> > > UnknownHostException.class, ConnectException.class,
>> > > > > > >> SSLException.class)
>> > > > > > >> > > - + Some other reasons
>> > > > > > >> > >
>> > > > > > >> > > See:
>> > > > > > >> > > https://github.com/apache/httpclient/blob/4.5.x/
>> > > > > > >> > > httpclient/src/main/java/org/apache/http/impl/client/
>> > > > > > >> > > DefaultHttpRequestRetryHandler.java#L129
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > Regards
>> > > > > > >> > > Philippe
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > On Wed, Mar 8, 2017 at 2:14 PM, Tuukka
Mustonen <
>> > > > > > >> > tuukka.mustonen@gmail.com <javascript:;>
>> > > > > > >> > > > wrote:
>> > > > > > >> > >
>> > > > > > >> > >> My problem is that AWS Application
Load Balancer (ALB)
>> > > > terminates
>> > > > > > all
>> > > > > > >> > >> existing connections during configuration
changes
>> > (including
>> > > > > > >> > >> auto-scaling).
>> > > > > > >> > >> As my perf tests trigger auto-scaling,
I want to retry
>> > failed
>> > > > > > requests
>> > > > > > >> > >> that
>> > > > > > >> > >> ALB connection termination causes.
>> > > > > > >> > >>
>> > > > > > >> > >> This results in a bunch of NoHttpResponseException
>> whenever
>> > > > > scaling
>> > > > > > >> > occurs
>> > > > > > >> > >> (=when ALB terminates existing connections).
>> > > > > > >> > >>
>> > > > > > >> > >> (And yeah, this load balancer behavior
is weird and ugly
>> > but
>> > > > it's
>> > > > > > what
>> > > > > > >> > >> they
>> > > > > > >> > >> confirmed).
>> > > > > > >> > >>
>> > > > > > >> > >> Using HttpClient 4, In user.properties
I have set:
>> > > > > > >> > >>
>> > > > > > >> > >> httpclient4.retrycount=1
>> > > > > > >> > >>
>> > > > > > >> > >> But that does nothing. I even tried:
>> > > > > > >> > >>
>> > > > > > >> > >> httpclient4.retrycount=100000000
>> > > > > > >> > >>
>> > > > > > >> > >> But zero effect.
>> > > > > > >> > >>
>> > > > > > >> > >> Switching to HttpClient 3.1 reproduces
the problem.
>> > However,
>> > > > with
>> > > > > > >> > >> HttpClient 3 and:
>> > > > > > >> > >>
>> > > > > > >> > >> httpclient3.retrycount=1
>> > > > > > >> > >>
>> > > > > > >> > >> The problem vanishes so I assume
retrying then works.
>> > > > > > >> > >>
>> > > > > > >> > >> I couldn't find where retry-attempts
are logged so I
>> have
>> > no
>> > > > > > "proof"
>> > > > > > >> > that
>> > > > > > >> > >> HttpClient 4 wouldn't actually retry,
but a) retrycount
>> > makes
>> > > > > > >> difference
>> > > > > > >> > >> on
>> > > > > > >> > >> HttpClient 3.1 but not on 4 b) I
would assume my whole
>> > > process
>> > > > > > should
>> > > > > > >> > >> crash
>> > > > > > >> > >> with retrycount=100000000 if it was
really applied.
>> > > > > > >> > >>
>> > > > > > >> > >> Related question: is http.connection.stalecheck$
>> > > Boolean=false
>> > > > > (in
>> > > > > > >> > >> hc.parameters file) valid anymore
on JMeter 3.x with
>> > improved
>> > > > > > >> > retry/stale
>> > > > > > >> > >> logic (and httpclient4.validate_after_inactivity)?
The
>> > line
>> > > is
>> > > > > > still
>> > > > > > >> > >> there
>> > > > > > >> > >> in bundled hc.parameters file...
>> > > > > > >> > >>
>> > > > > > >> > >> Tested on JMeter 3.1.
>> > > > > > >> > >>
>> > > > > > >> > >> Any open (or already fixed) tickets
about this? Couldn't
>> > find
>> > > > > > any...
>> > > > > > >> > >>
>> > > > > > >> > >> FYI: to describe better what happens
during ALB
>> connection
>> > > > > > >> termination:
>> > > > > > >> > >>
>> > > > > > >> > >> ...Successful communication with
keep-alive...
>> > > > > > >> > >> - ALB responds to a request, and
sends Connection:
>> > keep-alive
>> > > > so
>> > > > > > >> JMeter
>> > > > > > >> > >> leaves the connection open
>> > > > > > >> > >> - JMeter sends a new request
>> > > > > > >> > >> - ALB may might or might not ack
it (not sure if there
>> was
>> > > > > pattern)
>> > > > > > >> > >> - ALB closes connection on TCP level
(FIN)
>> > > > > > >> > >> - Connection gets closed and so sent
request failed and
>> > needs
>> > > > to
>> > > > > be
>> > > > > > >> > >> retried
>> > > > > > >> > >>
>> > > > > > >> > >> I think this generates NoHttpResponseException
in
>> JMeter.
>> > > > > > >> > >>
>> > > > > > >> > >> Tuukka
>> > > > > > >> > >>
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > --
>> > > > > > >> > > Cordialement.
>> > > > > > >> > > Philippe Mouawad.
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >> >
>> > > > > > >> > --
>> > > > > > >> > Cordialement.
>> > > > > > >> > Philippe Mouawad.
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Cordialement.
>> > > > > > > Philippe Mouawad.
>> > > > > >
>> > > > > > ------------------------------------------------------------
>> > > ---------
>> > > > > > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > > > > > For additional commands, e-mail: user-help@jmeter.apache.org
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Cordialement.
>> > > > > Philippe Mouawad.
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Cordialement.
>> > > Philippe Mouawad.
>> > >
>> >
>>
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message