jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Devaney <bdeva...@docfinity.com>
Subject Re: Random NoHttpResponseException
Date Thu, 30 Oct 2014 14:31:05 GMT
The problem is definitely the load balancer.  I added some more logging and
monitoring and found the cause of the problem with JMeter.  It does not
happen at the same time every night, it's completely random and occurred 5
times last night.  A segmentation fault is reported in the load balancer
error log at precisely the second after JMeter sufferes its one second
error.  Now I just need to track down what is causing the segmentation
fault.

Thanks for all your help.


On Wed, Oct 29, 2014 at 7:03 PM, Shmuel Krakower <shmulikk@gmail.com> wrote:

> Sounds like a problem with your load balancer.
>
> 1. Is it the same timestamp every night or happen only once per night,
> every night on a different time?
> 2. Do you have any error logs or monitoring on your load balancer?
> 3. Do you have any other metrics on the load balancer?
> 4. Could you try running similar tests in parallel from another host and
> see if both tests result with this behavior on the exact time frame?
>
> Best
>
> www.beatsoo.org - free application performance monitoring from world wide
> locations.
> On Oct 29, 2014 10:31 PM, "Brian Devaney" <bdevaney@docfinity.com> wrote:
>
> > I am not sure that this is even a JMeter problem, but I will post anyway
> > because JMeter seems to be the only way I can cause and detect the
> problem.
> >
> > I have a number of JMeter scripts that get run in a test overnight.  Each
> > script is run three times with increasing numbers of users each time.
> The
> > script is hitting a cluster of two nodes with a load balancer in front.
> > The scripts run perfectly over the course of the night, except in one or
> > two cases, a number of errors will occur, which is where I get the
> > NoHttpResponseException.  At what seems to be a completely random moment,
> > for one second, all requests that are getting sent to the load balancer
> > will get this error.  According to the jmeter.log it always seems to be
> > within one second.  It will not, for example, start halfway through
> > 10:05:19 and finish halfway through 10:05:20.  Every log entry with the
> > error will be within the same second of 10:05:19.
> >
> > The stack trace that gets generated from one of these errors is:
> > org.apache.commons.httpclient.NoHttpResponseException: The server
> > 10.10.2.179 failed to respond
> >     at
> >
> >
> org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1976)
> >     at
> >
> >
> org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> >     at
> >
> >
> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> >     at
> >
> >
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> >     at
> >
> >
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> >     at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> >     at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> >     at
> >
> >
> org.apache.jmeter.protocol.http.sampler.SoapSampler.sample(SoapSampler.java:272)
> >     at
> >
> >
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1105)
> >     at
> >
> >
> org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1094)
> >     at
> >
> >
> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
> >     at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
> >     at java.lang.Thread.run(Thread.java:722)
> >
> >
> > As part of my testing, I have turned off any timeouts in JMeter that I
> can,
> > or made them long enough that I will get a response eventually.  Both
> nodes
> > in the cluster, the load balancer and the server running the JMeter test
> > all have ntp installed in order to keep their clocks in sync.  As a test
> I
> > ran one of my JMeter scripts while monitoring the time drift to see if
> the
> > clock getting reset during the test could be the culprit.  The exception
> > did not show up on that run.  Another possibility was that the cluster
> > could not handle the load, but in my tests, I have seen the case where a
> > script with 300 users has the error and the same script with 500 users
> > works perfectly with no configuration change other than the number of
> users
> > running the script.
> >
> > Like I said, this may be outside of a JMeter problem, but if anyone has
> any
> > ideas, I would greatly appreciate it.  Thank you.
> >
> > --
> >
> > --
> > Brian Devaney
> > DocFinity® by Optical Image Technology, Inc.
> > 100 Oakwood Avenue, State College,  PA  16803
> > ph: 814.238.0006 ext. 274
> > fax: 814.238.0011
> > mailto:bdevaney@DocFinity.com
> > http://www.docfinity.com/
> >
> >
> > DISCLAIMER: This email may contain proprietary information, some or all
> > of which may be legally privileged. It is for the intended recipient
> > only. If an addressing or transmission error has misdirected this email,
> > please notify the author by replying to this email. If you are not the
> > intended recipient, you may not use, disclose, distribute, copy, print,
> > or rely on this email.
> >
>



-- 

-- 
Brian Devaney
DocFinity® by Optical Image Technology, Inc.
100 Oakwood Avenue, State College,  PA  16803
ph: 814.238.0006 ext. 274
fax: 814.238.0011
mailto:bdevaney@DocFinity.com
http://www.docfinity.com/


DISCLAIMER: This email may contain proprietary information, some or all
of which may be legally privileged. It is for the intended recipient
only. If an addressing or transmission error has misdirected this email,
please notify the author by replying to this email. If you are not the
intended recipient, you may not use, disclose, distribute, copy, print,
or rely on this email.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message