hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hiranya Jayathilaka <hiranya...@gmail.com>
Subject Re: Migrating Synapse to Latest HTTP Core
Date Wed, 24 Jul 2013 08:36:40 GMT
Hi Oleg,

I can reproduce the problem with the attached JUnit test case. This test case doesn't use
any Axis2/Synapse related code. It only requires HTTP Core and HTTP client. I used HTTP Core
4.2.4 for testing.

The test case starts a ListeningIOReactor, does a simple HTTP request-response exchange and
then terminates the reactor. Then after a couple of seconds, it tries to start the IO reactor
again and the exception occurs. Here's the full output:

Round 1
Listener started on port: 8280
Connected
Request received
Response submitted
Waiting - Shutdown in 5 secs
Shutdown

… wait 2 seconds...

Round 2
Listener started on port: 8280
Waiting - Shutdown in 5 secs
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
	at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
	at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
	at org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101)
Shutdown

Thanks,
Hiranya



On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <olegk@apache.org> wrote:

> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
>> It looks like the socket goes into the CLOSE_WAIT state after shutting
>> down the IO reactor and remains there until the JVM it self is killed.
>> Since all automated test cases are executed within a single JVM
>> instance, this issue continues to occur. Looks like something is not
>> getting cleaned up properly with the latest HTTP Core.
>> 
>> 
>> Thanks,
>> Hiranya
>> 
> 
> Hiranya
> 
> I need a more specific evidence of HttpCore doing something wrong. Could
> you please reproduce the issue with a test app, preferably not dependent
> on Synapse?
> 
> Oleg   
> 
>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
>> <hiranya911@gmail.com> wrote:
>> 
>>> Hi Oleg,
>>> 
>>> 
>>> I've done the necessary code changes locally and it seems to work
>>> fine. The HTTPS sender now works as expected. But when running
>>> automated tests, I frequently get the following exception:
>>> 
>>> 
>>> 34 [HttpCoreNIOListener] WARN
>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
>>> be unstable: IOReactor encountered a checked exception : Address
>>> already in use
>>> java.net.BindException: Address already in use
>>>        at sun.nio.ch.Net.bind(Native Method)
>>>        at
>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
>>>        at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>>        at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
>>>        at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
>>>        at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
>>>        at
>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
>>>        at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
>>> $2.run(HttpCoreNIOListener.java:254)
>>>        at java.lang.Thread.run(Thread.java:680)
>>> 
>>> 
>>> This is seen in both NHTTP unit tests and Synapse integration tests.
>>> Any idea what is going on here? Synapse cleans up the IO reactor by
>>> calling ioReactor.shutdown(). Is there anything else we need to do
>>> to clean up the reactor properly?
>>> 
>>> 
>>> Thanks,
>>> Hiranya
>>> 
>>> 
>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
>>> <hiranya911@gmail.com> wrote:
>>> 
>>>> HI Oleg,
>>>> 
>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <olegk@apache.org>
>>>> wrote:
>>>> 
>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
>>>>>> <rajika.kumarasiri@gmail.com> wrote:
>>>>>> 
>>>>>>> May be we should enable the SSl debug logs and see what's
>>>>>>> going on ? 
>>>>>> 
>>>>>> I ran some more tests and it looks like the HTTPS sender
>>>>>> doesn't work with HTTP Core 4.2. It just sends out the
>>>>>> messages without any SSL security. I can even monitor the
>>>>>> message in plain text using TCPMon. Looks like some SSL
>>>>>> related code in the Synapse HTTPS sender doesn't work properly
>>>>>> with the new HTTP Core.
>>>>>> 
>>>>>> Thanks,
>>>>>> Hiranya
>>>>>> 
>>>>> 
>>>>> Hiranya
>>>>> 
>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs? 
>>>> 
>>>> 
>>>> No not yet. I just tried changing the HTTP Core dependency version
>>>> to begin with. I guess migrating off the deprecated APIs is the
>>>> next logical step.
>>>> 
>>>> 
>>>> Thanks,
>>>> Hiranya
>>>> 
>>>>> 
>>>>> Oleg
>>>>> 
>>>>> PS: By the way, you should probably consider migrating straight
>>>>> to 4.3
>>>>> at this point.
>>>>> 
>>>>> 
>>>>>>> 
>>>>>>> Rajika
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
>>>>>>> <hiranya911@gmail.com> wrote:
>>>>>>> Hi Folks,
>>>>>>> 
>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried
>>>>>>> using 4.2 instead (just changed the version of the maven
>>>>>>> dependency). Almost everything worked fine, except for a
>>>>>>> couple of integration test failures. In both these tests
>>>>>>> Synapse uses HTTPS to contact the backend services, so I
>>>>>>> suppose that's where things went wrong. The backend server
>>>>>>> threw the following exception in the process:
>>>>>>> 
>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O
>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
>>>>>>> SSL message, plaintext connection?
>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message,
>>>>>>> plaintext connection?
>>>>>>>       at
>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>>>>       at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>>>>       at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>>>>       at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>>>>       at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>>>>       at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>>>>       at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>>>>       at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>>>>       at java.lang.Thread.run(Thread.java:680)
>>>>>>> 
>>>>>>> I also tried upgrading to the latest HTTP Core version
>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most
>>>>>>> of the integration tests failed too. It turns out only the
>>>>>>> very first integration test ran successfully. Everything
>>>>>>> that followed threw a bind exception. Looks like something
>>>>>>> isn't getting cleaned up properly (although the logs
>>>>>>> indicate that Synapse transport listeners are shutting down
>>>>>>> cleanly after each test case).
>>>>>>> 
>>>>>>> Does anybody got an idea what's going on? Do we need to do
>>>>>>> any code changes to migrate to the latest HTTP Core?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>>>>>>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Hiranya Jayathilaka
>>>>>> Mayhem Lab/RACE Lab;
>>>>>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>>>>>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>>> 
>>>> 
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>> 
>>>> 
>>> 
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>> 
>>> 
>> 
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>> 
>> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
> 

--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB;  http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu;  Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com


Mime
View raw message