Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Received-SPF: unknown ipv4:213.46.255.15/27 (athena.apache.org: encountered
 unrecognized mechanism during SPF processing of domain of odi@odi.ch)
Message-ID: <4A36CE22.9040804@odi.ch>
Date: Tue, 16 Jun 2009 00:41:38 +0200
From: =?ISO-8859-1?Q?Ortwin_Gl=FCck?= <odi@odi.ch>
User-Agent: Thunderbird 2.0.0.21 (X11/20090321)
MIME-Version: 1.0
To: HttpComponents Project <dev@hc.apache.org>
Subject: Re: Question on HTTPClient-675
References: <f97c4b410906150957k107ccc3s79fd475ad3f4afe5@mail.gmail.com>
	 <4A368361.9090405@odi.ch>
	 <f97c4b410906151039h5b9bcda4p747593dfc209f8fe@mail.gmail.com>
	 <4A369EE4.8030409@odi.ch>
 <f97c4b410906151311h7af15979r1020d04797023598@mail.gmail.com>
In-Reply-To: <f97c4b410906151311h7af15979r1020d04797023598@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


D H wrote:
> 
> I agree about this, they wanted proof of a fix before any changes would be
> made and my manager is still saying there must be proof before code changes
> even now after I showed him your email and the documentation from the site.

Nice way of thinking -- at a university where time and money are infinite 
resources. Wishful thinking anywhere else. Enter The Real World.

>> Anyway if this is so hard to reproduce and profiling didn't give you any
>> idea what's broken, why do you suspect something in this particular piece of
>> code is the cause of your problem?
> 
> 
> That's a very good question, I was given the source code, no explanation of
> their code and told HttpClient was the problem.  

Why am I not convinced? But it's well possible. My best advice I can give is: 
Upgrade for God's sake, and fix obvious mistakes in the use of the API. Use some 
monitoring tools like JConsole / JVisualVM, jmap, netstat, top and the like to 
see if you have a garbage collection problem or any obvious resource leaks, then 
take appropriate action.

 > It is a rather large
> codebase so I'm taking their word for it.  They've apparently worked on it
> sporadically for a couple of months to isolate it to HttpClient, and my task
> is to prove it is causing this problem with JMeter in a self-contained code
> sample.  This problem has only been seen in Production and only after almost
> a week of running 24/7 so it's hard to duplicate it easily.  I've sent over
> two hundred thousand HttpClient requests without seeing the problem so I'd
> rather see this code fix go to Production and test that way personally.

Sounds like a a typical production problem to me: it can take weeks to see it, 
there is no way you can trigger it on purpose and maybe it has only ever been 
encountered on the production system.

Face it, you will not reproduce it locally in reasonable time. It is maybe 
dependent on the workload you are running. Maybe it's even platform specific and 
may not trigger on your testing platform. And with platform I don't mean just 
the OS. Also the processor type (Single core, Multi core) can make a huge 
difference.

What can really help you is to expect the situation in production. And instead 
of panicing and quickly restarting, take your time, having the right tools at 
hand to find out what's going on in this moment. Maybe even a "post mortem" job 
that gathers useful information in case this happens during everyone is 
sleeping. Is it swapping? Has the VM run out of memory (stack, heap, perm gen, 
code) and is constantly GCing? Has the OS run out of file descriptors? To which 
signals does it react? Is it creating threads at a high rate? Are there just too 
many runnable threads? Is it busy waiting? Is it looping endlessly? Is it I/O 
bound? Is it lock contented or even deadlocked? Is it blocking on I/O or 
network? Is it waiting for the DB? What's going on on the DB? What's going on on 
the network? Is your log detailes enough to give you the information you need?

If all fails you, you may have to live with it and rather setup monitoring 
infrastructure that can reliably detect the situation and restart the process.

> 
> I really appreciate you taking the time to answer my emails, thank you very
> much.
> 
> Sincerely,
> David Hamilton

Ortwin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org