Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Received-SPF: neutral (herse.apache.org: local policy)
Message-ID: <4541055E.8010603@hanik.com>
Date: Thu, 26 Oct 2006 13:58:38 -0500
From: Filip Hanik - Dev Lists <devlists@hanik.com>
User-Agent: Thunderbird 1.5.0.7 (Windows/20060909)
MIME-Version: 1.0
To: Tomcat Developers List <dev@tomcat.apache.org>
Subject: Re: svn commit: r467787 - in
 /tomcat/tc6.0.x/trunk/java/org/apache/tomcat/util/net:
 NioChannel.java NioEndpoint.java	 SecureNioChannel.java SocketProperties.java
References: <20061025221111.7490D1A984D@eris.apache.org>
 <45408281.4040706@apache.org> <4540D774.8080804@hanik.com>
 <4540FBB9.5070107@kippdata.de>
In-Reply-To: <4540FBB9.5070107@kippdata.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

That's some very good info, it looks like my system never does go over 
30k and cleaning it up seems to be working really well.
btw. do you know where I change the cleanup intervals for linux 2.6 kernel?

I figured out what the problem was:
Somewhere I have a lock/wait problem

for example, this runs perfectly:
./ab -n 1 -c 100 http://localhost:$PORT/run.jsp?run=TEST$i

If I change -c 100 (100 sockets) to -c 1, each JSP request takes 1 second.

so what was happening in my test was running 1000 requests over 400 
connections, then invoking 1 request over 1 connection, and repeat.
Every time I did the single connection request, it does a 1sec delay, 
this cause the CPU to drop.

So basically, the NIO connector sucks majorly if you are a single user 
:), I'll trace this one down.
Filip


Rainer Jung wrote:
> Hi Filip,
>
> the fluctuation reminds me of something: depending on the client
> behaviour connections will end up in TIME_WAIT state. Usually you run
> into trouble (throughput stalls) once you have around 30K of them. They
> will be cleaned up every now and then by the kernel (talking about the
> unix/Linux style mechanisms) and then throughput (and CPU usage) start
> again.
>
> With modern systems handling 10-20k requests per second one can run into
> trouble much faster, than the usual cleanup intervals.
>
> Check with "netstat -an" if you can see a lot of TIME_WAIT connections
> (thousands). If not it's something different :(
>
> Regards,
>
> Rainer
>
> Filip Hanik - Dev Lists schrieb:
>   
>> Remy Maucherat wrote:
>>     
>>> fhanik@apache.org wrote:
>>>       
>>>> Author: fhanik
>>>> Date: Wed Oct 25 15:11:10 2006
>>>> New Revision: 467787
>>>>
>>>> URL: http://svn.apache.org/viewvc?view=rev&rev=467787
>>>> Log:
>>>> Documented socket properties
>>>> Added in the ability to cache bytebuffers based on number of channels
>>>> or number of bytes
>>>> Added in nonGC poller events to lower CPU usage during high traffic 
>>>>         
>>> I'm starting to get emails again, so sorry for not replying.
>>>
>>> I am testing with the default VM settings, which basically means that
>>> excessive GC will have a very visible impact. I am testing to
>>> optimize, not to see which connector would be faster in the real world
>>> (probably neither unless testing scalability), so I think it's
>>> reasonable.
>>>
>>> This fixes the paranormal behavior I was seeing on Windows, so the NIO
>>> connector works properly now. Great ! However, I still have NIO which
>>> is slower than java.io which is slower than APR. It's ok if some
>>> solutions are better than others on certain platforms of course.
>>>
>>>       
>> thanks for the feedback, I'm testing with larger files now, 100k+ and
>> also see APR->JIO->NIO
>> NIO has a very funny CPU telemetry graph, it fluctuates way to much, so
>> I have to find where in the code it would do this, so there is still
>> some work to do.
>> I'd like to see a nearly flat CPU usage when running my test, but
>> instead the CPU goes from 20-80% up and down, up and down.
>>
>> during my test
>> (for i in $(seq 1 100); do echo -n "$i."; ./ab -n 1000 -c 400
>> http://localhost:$PORT/104k.jpg 2>1 |grep "Requests per"; done)
>>
>> my memory usage goes up to 40MB, then after a FullGC it goes down to
>> 10MB again, so I wanna figure out where that comes from as well. My
>> guess is that all that data is actually in the java.net.Socket classes,
>> as I am seeing the same results with the JIO connector, but not with
>> APR(cause APR allocates mem using pools)
>> Btw, had to put in the byte[] buffer back into the
>> InternalNioOutputBuffer.java, ByteBuffers are way to slow.
>>
>> With APR, I think the connections might be lingering to long as
>> eventually, during my test, it stop accepting connections. Usually
>> around the 89th iteration of the test.
>> I'm gonna keep working on this for a bit, as I think I am getting to a
>> point with the NIO connector where it is a viable alternative.
>>
>> Filip
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org