Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <48B30C8F.9060609@kippdata.de>
Date: Mon, 25 Aug 2008 21:48:31 +0200
From: Rainer Jung <rainer.jung@kippdata.de>
User-Agent: Thunderbird 2.0.0.16 (Windows/20080708)
MIME-Version: 1.0
To: Tomcat Developers List <dev@tomcat.apache.org>
Subject: Re: 5.5.27 blocker: URIEncoding UTF-8 broken for 5.5.trunk
References: <48B2CCBD.3040708@kippdata.de>
 <1219679185.3032.38.camel@localhost.localdomain>
 <48B2D5D6.5080203@kippdata.de>				 <48B2DD00.20405@hanik.com>
 <48B2DEB4.7090002@kippdata.de> <48B2EFBE.60205@kippdata.de>
 <48B2F58B.1040706@kippdata.de> <48B3050B.4000103@kippdata.de>
In-Reply-To: <48B3050B.4000103@kippdata.de>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

I need a small correction: when I delete the available method of the
IntermediateInputStream, I can neither reproduce the redirect problem,
nor bz44494. My test against bz44494 was wrong.

Nevertheless I would strongly prefer the removal of conv.recycle in
convertURI in the Adapter code, because I think I understand much
better, that we don't need this, than I understand the exact need for
the available() method.

BTW: The conv.recycle in convertURI is there since the method convertURI
was created. At that time recycle was an empty method though. I think we
now gave it a meaning, which is not right for both use cases of
B2CConverter.

I didn't test it, but I expect TC 4.1 trunk to be broken in the same way.

Concerning tc6.0 and trunk: we should remove conv.recycle() from
convertURI there as well and whoever knows, why we have the available()
method should comment, if we need to introduce it to 6.0 and trunk too.

Regards,

Rainer

Rainer Jung schrieb:
> Unfortunately removing the available() method agains gives us back
> 44494. But there is some hope.
> 
> B2CConverter gets used for two different things:
> 
> 1) It is associated with a request and used inside the adapter to decode
> the URI.
> 
> In this case, it seems that during the lifetime of the  ByteChunk
> underlying the B2CConverter (the URI ByteChunk) we only call
> conv.convert() once. So there is no real need for any conv.recycle()
> before conv.convert() (cleaning up some unrelated old ByteChunk; if we
> would need to do that, then we should do it at the end of convertURI for
> some request and not near the beginning of convertURI for some later
> request).
> 
> 2) It is associated with the InputBuffer to transform input data.
> 
> In this case we mightcall convert() multiple times and handle a
> difficult relationship between direct access to the Chunks and via
> convert. Here we need to clean up the ByteChunk at the end of the
> request handling.
> 
> As far as I can see, the instances of B2CConverter used for 1) and for
> 2) are independant of each other.
> 
> B2CConvertor.recycle() has only one action, it calls
> ReadConvertor.recycle(), which reads the remaining bytes in the
> ByteChunk() and throws them away. This seems to be good for 2) (e.g. BZ
> 44494), but it eats data from a ByteChunk instance associated with the
> B2CConverter during an earlier request in case 1).
> 
> I would suggest to simply remove the conv.recycle in case 1). I dn't see
> any use of it. If we really think we need to eat up remaining bytes in
> the ByteChunk in 1), then we should do the conv.recycle() near the end
> of convertURI.
> 
> Regards,
> 
> Rainer
> 
> Rainer Jung schrieb:
>> It's the available() methde of the class IntermediateInputStream
>> contained in B2CConverter. It doesn't exist in 6.0. If I comment it out
>> in 5.5 trunk. the problem is gone.
>>
>> The method was first introduced in
>>
>> http://svn.apache.org/viewvc/tomcat/connectors/trunk/util/java/org/apache/tomcat/util/buf/B2CConverter.java?r1=481614&r2=568699&pathrev=568699&diff_format=h
>>
>> and the changed to it's final contents in
>>
>> http://svn.apache.org/viewvc/tomcat/connectors/trunk/util/java/org/apache/tomcat/util/buf/B2CConverter.java?r1=568699&r2=569970&diff_format=h
>>
>> I didn't yet check, what negative consequences removing of the method
>> has, or if there is a better implementation for it. We should carefully
>> check the consequences of changing it w.r.t. BZ 44494 with the test
>> webapp, we got for that BZ.
>>
>> Regards,
>>
>> Rainer
>>
>>
>> Rainer Jung schrieb:
>>> OK, cancelled my appointment. More info:
>>>
>>> I backported all functional changes in o.a.tomcat.util.buf from tc6.x to
>>> tc5.5 and can't reproduce the problem any more. Those are very few
>>> changes. I'll narrow it down some more during the next hour. Stay tuned.
>>>
>>> Rainer
>>>
>>> Rainer Jung schrieb:
>>>> Filip Hanik - Dev Lists schrieb:
>>>>> Thanks Rainer, I will take a look at it tonight
>>>> Thank you!
>>>>
>>>> Last info chunk for today: in CoyoteAdapter.convertURI, before the
>>>> try/catch block that either creates or recycles the B2CConverter, the
>>>> ByteChunk bc coming from the decodedURI contains the correct URI. After
>>>> the recycle of the B2CConverter, the ByteChunk is empty and thus we end
>>>> up in a default redirect.
>>>>
>>>> Although we already read the correct URI from the request, the
>>>> B2CConverter associated with the request detroys the already read URI in
>>>> the recycle. I don't see the delta to 6.0. CoyoteAdapter seems fine,
>>>> maybe in ByteChunk?
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>> Filip
>>>>>
>>>>> Rainer Jung wrote:
>>>>>> Remy Maucherat schrieb:
>>>>>>  
>>>>>>> On Mon, 2008-08-25 at 17:16 +0200, Rainer Jung wrote:
>>>>>>>    
>>>>>>>> If we revert the backport of
>>>>>>>>
>>>>>>>> http://svn.eu.apache.org/viewvc/tomcat/trunk/java/org/apache/tomcat/util/buf/B2CConverter.java?r1=642819&r2=647307&diff_format=h
>>>>>>>>
>>>>>>>>
>>>>>>>> then the redirect loop is gone, and the usual content gets served, but
>>>>>>>> we know, that this change was needed to fix the "remaining garbage"
>>>>>>>> part
>>>>>>>> of 44494. So reverting it without any alternative is not really an
>>>>>>>> option.
>>>>>>>>       
>>>>>>> Most likely, some backport is missing (many patches made up this input
>>>>>>> fixes). OTOH, I don't think this particular problem is that critical, so
>>>>>>> I would be in favor of dropping it if fixing the issue is complex.
>>>>>>>     
>>>>>> Got some more info:
>>>>>>
>>>>>> 1) What happens inside the new ReadConvertor.recycle(): actually if I
>>>>>> print out, which "superfluous" bytes are eaten by the read() loop inside
>>>>>> ReadConvertor.recycle(), I can see, that it's all the bytes making up
>>>>>> the PATH in the URL. If I request http://myserver:8080/, recycle read
>>>>>> one character, namely "/", if I request http://myserver:8080/index.jsp,
>>>>>> then recycle reads all characters from "/index.jsp". In my understanding
>>>>>> of those patches, the time recycle in ReadConvertor gets called, those
>>>>>> should have alrady been read and only body bytes left over after request
>>>>>> processing should be eaten.
>>>>>>
>>>>>> 2) So I checked, when recycle() gets called, and I see, that during the
>>>>>> first few (here: 2) requests it doesn't get called at all (and those
>>>>>> work), and during the following broken requests, it gets called in the
>>>>>> following stack:
>>>>>>
>>>>>>         at
>>>>>> org.apache.tomcat.util.buf.ReadConvertor.recycle(B2CConverter.java:222)
>>>>>>         at
>>>>>> org.apache.tomcat.util.buf.B2CConverter.recycle(B2CConverter.java:64)
>>>>>>         at
>>>>>> org.apache.catalina.connector.CoyoteAdapter.convertURI(CoyoteAdapter.java:475)
>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:265)
>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:172)
>>>>>>
>>>>>>         at
>>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
>>>>>>
>>>>>>         at
>>>>>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>>>>>>
>>>>>>         at
>>>>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>>>>>>
>>>>>>         at
>>>>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>>>>>>
>>>>>>         at
>>>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:595)
>>>>>>
>>>>>> Didn't yet go further into it, but maybe something is wrong in
>>>>>> CoyoteAdapter.convertURI?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>> P.S: I'll soon need to stop investigating this for today. If anyone can
>>>>>> take over that will be nice, because we really should have a working
>>>>>> 5.5.27 soon.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org