axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vineeth Narayanan (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (AXIS2-2883) CLOSE_WAIT slowly building up over the period of time.
Date Mon, 22 Feb 2010 22:08:28 GMT

    [ https://issues.apache.org/jira/browse/AXIS2-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836938#action_12836938
] 

Vineeth Narayanan edited comment on AXIS2-2883 at 2/22/10 10:07 PM:
--------------------------------------------------------------------

Hi,

We are trying to upgrade from Axis2 1.4 to Axis2 1.5.1. We are using autogenerated stub from
the wsdl using the wsdl2java maven plugin. Our code was running fine using 1.4 version but
since upgrading to 1.5.1 we are facing a couple of issues. the approach which we had taken
with 1.4 was to create a context once for the object in the constructor using a custom axis2.xml(though
this is the same as the axis2.xml which comes with the Axis2 distribution. 

                context = ConfigurationContextFactory.createBasicConfigurationContext("axis2.xml");

We create a stub once using this context and then reuse this stub throughout the life cycle.
While initializing the stub, we create an instance of the multithreadedHttpConnectionmanager
and set its properties using the params object. we create a httpclient using this manager
instance and cache the client in the context with client reuse flag set to true.

                MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
                HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
                // Maximum one socket connection to a specific host
                connectionManagerParams.setDefaultMaxConnectionsPerHost(1);
                connectionManagerParams.setTcpNoDelay(true);
                connectionManagerParams.setStaleCheckingEnabled(true);
                connectionManagerParams.setLinger(0);
                connectionManager.setParams(connectionManagerParams);
                HttpClient httpClient = new HttpClient(connectionManager);
                context.setProperty(HTTPConstants.REUSE_HTTP_CLIENT,
                    Boolean.TRUE);
                context.setProperty(HTTPConstants.CACHED_HTTP_CLIENT,
                    httpClient);

                stub = new SoapStub(context,endPoint);

The endPoint is passed as a parameter to the constructor. We then use the stub to call the
actual service method to get the response and call the cleanup method of the stub in the finally
block. the stub code uses the operationClinet.execute(true) method. All this was runnig fine
in 1.4. since migrating to 1.5.1 we first faced the issue where our server just hung after
a couple of request as it was uanble to get a connection from the webservice. To overcome
this I have tried to create a stub for each request. as well as creating a context for each
request. The problem which we now face is that after about 100 requests we get the following
error

2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.axis2.transport.http.HTTPSender - Unable to sendViaPost
to url[http://uat.cib.intranet.com/server_dapdev/Soap]
java.net.BindException: Address already in use: connect
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:525)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
	at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
	at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
	at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
	at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
	at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
	at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
	at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
	at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
	at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)

We have tried all possible combinations which  have been posted on different forums including
setting the Options object but to no avail. We have tried calling the cleanuptransport method
as well as calling the httpMehtod.releaseConnection & the Connectionmanager.closeIdleConnections
in the finally block. But none of them resolve this issue. We cannot call Connectionmanager.shutdown
as we then get the factory is shutdown error in subsequent calls. If we put a gc call in the
finally block then all runs fine, but this slows up our application response time drastically.


Is anybody else facing the same issue since migrating to 1.5.1? 

Since there is an inherent fix in 1.5.1 for the client reuse and CLOSE_WAIT issue, is there
something that I need to change in my code? I am a newbie to using Axis2 for webservice.

I am sorry if this is not the correct forum to post this but I have exhausted all possible
options to resolve this issue but am having no success. 

      was (Author: vineeth):
    Hi,

We are trying to upgrade from Axis2 1.4 to Axis2 1.5.1. We are using autogenerated stub from
the wsdl using the wsdl2java maven plugin. Our code was running fine using 1.4 version but
since upgrading to 1.5.1 we are facing a couple of issues. the approach which we had taken
with 1.4 was to create a context once for the object in the constructor using a custom axis2.xml(though
this is the same as the axis2.xml which comes with the Axis2 distribution. 

                context = ConfigurationContextFactory.createBasicConfigurationContext("axis2.xml");

We create a stub once using this context and then reuse this stub throughout the life cycle.
While initializing the stub, we create an instance of the multithreadedHttpConnectionmanager
and set its properties using the params object. we create a httpclient using this manager
instance and cache the client in the context with client reuse flag set to true.

                MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
                HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
                // Maximum one socket connection to a specific host
                connectionManagerParams.setDefaultMaxConnectionsPerHost(1);
                connectionManagerParams.setTcpNoDelay(true);
                connectionManagerParams.setStaleCheckingEnabled(true);
                connectionManagerParams.setLinger(0);
                connectionManager.setParams(connectionManagerParams);
                HttpClient httpClient = new HttpClient(connectionManager);
                context.setProperty(HTTPConstants.REUSE_HTTP_CLIENT,
                    Boolean.TRUE);
                context.setProperty(HTTPConstants.CACHED_HTTP_CLIENT,
                    httpClient);

                stub = new SoapStub(context,endPoint);

The endPoint is passed as a parameter to the constructor. We then use the stub to call the
actual service method to get the response and call the cleanup method of the stub in the finally
block. the stub code uses the operationClinet.execute(true) method. All this was runnig fine
in 1.4. since migrating to 1.5.1 we first faced the issue where our server just hung after
a couple of request as it was uanble to get a connection from the webservice. To overcome
this I have tried to create a stub for each request. as well as creating a context for each
request. The problem which we now face is that after about 100 requests we get the following
error

2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - I/O exception
(java.net.BindException) caught when processing request: Address already in use: connect
2010-02-22 13:55:19,734 INFO  org.apache.commons.httpclient.HttpMethodDirector - Retrying
request
2010-02-22 13:55:19,734 INFO  org.apache.axis2.transport.http.HTTPSender - Unable to sendViaPost
to url[http://uat.mds.gm.cib.intranet.db.com/mds_server_dapdev/MDSSoap]
java.net.BindException: Address already in use: connect
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:525)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
	at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
	at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
	at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
	at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
	at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
	at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
	at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
	at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
	at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)

We have tried all possible combinations which  have been posted on different forums including
setting the Options object but to no avail. We have tried calling the cleanuptransport method
as well as calling the httpMehtod.releaseConnection & the Connectionmanager.closeIdleConnections
in the finally block. But none of them resolve this issue. We cannot call Connectionmanager.shutdown
as we then get the factory is shutdown error in subsequent calls. If we put a gc call in the
finally block then all runs fine, but this slows up our application response time drastically.


Is anybody else facing the same issue since migrating to 1.5.1? 

Since there is an inherent fix in 1.5.1 for the client reuse and CLOSE_WAIT issue, is there
something that I need to change in my code? I am a newbie to using Axis2 for webservice.

I am sorry if this is not the correct forum to post this but I have exhausted all possible
options to resolve this issue but am having no success. 
  
> CLOSE_WAIT slowly building up over the period of time.
> ------------------------------------------------------
>
>                 Key: AXIS2-2883
>                 URL: https://issues.apache.org/jira/browse/AXIS2-2883
>             Project: Axis2
>          Issue Type: Bug
>          Components: client-api
>    Affects Versions: 1.1
>         Environment: Operating System : Solaris 
> Axis2 Version : 1.1
> Application Server : weblogic 8.1 SP6
> Using with Cocoon and weblogic DSP
>            Reporter: Lakshmanan Venkatachalam
>            Assignee: Glen Daniels
>            Priority: Critical
>             Fix For: 1.5.1
>
>
> I am experiencing theconstant increase in close wait in the production environment over
the period 7 days. 
> We are using Synchronous webservices and we are calling two webservices 24 times every
day. We have allocated the maximum of 1.5 GB per application instance and we have two application
instances. We are utilizing maximum of 250 - 300 MB in average. So Full GC never runs in our
environment.
> It seems like the client API ServiceClient.java is not cleaning up the resources associated
with this component. We are creating the new ServiceClient component on every call we have
for webservices. Though we have called the cleanup() method at the end of every call to the
webservices. At times its not getting executed.
> But when we force garabage collection from the application, it was able to clear all
the CLOSE_WAIT components. Since we have similar cleanup() call on finalize() method, it is
able to do proper clean up when GC is collecting these objects.
> Forcing GC cannot be a solution, I like to hear from axis2 experts on how we can resolve
this problem properly and what could be the cause for this happening.
> Below is our client code for your reference.
> private WebServiceResponse invokeWebservice(OMElement inputElement,
> 			Options options) throws WebServiceInvokerException {
> 		ServiceClient serviceClient = null;
> 		try {
> 			serviceClient = new ServiceClient();
> 			serviceClient.setOptions(options);
> 			// This following line of code is used when we are using
> 			// WS-Addressing. User has to make sure the addressing MAR file in
> 			// class path before enable the following line of code
> 			//
> 			// serviceClient.engageModule(new QName(
> 			// org.apache.axis2.Constants.MODULE_ADDRESSING));
> 			// Invoking synchrounous webservice
> 			//
> 			OMElement result = serviceClient.sendReceive(inputElement);
> 			
> 			OMNode firstOMChild = result.getFirstOMChild();
> 			// Conver the OMelements to XML String
> 			//
> 			Writer stringWriter = new StringWriter();
> 			firstOMChild.serialize(stringWriter);
>             serviceClient.cleanup();
> 			stringWriter.flush();
> 			// Return the Axis2WebserviceResponse
> 			//
> 			return new Axis2WebServiceResponse(stringWriter.toString());
> 		} catch (AxisFault afe) {
> 			throw new WebServiceInvokerException(afe);
> 		} catch (XMLStreamException xse) {
> 			throw new WebServiceInvokerException(xse);
> 		} catch (IOException ioe) {
> 			throw new WebServiceInvokerException(ioe);
> 		} finally {
> 			try {
> 				serviceClient.cleanup();
>                 serviceClient=null;
> 			} catch (AxisFault axisFault) {
> 				//
> 			}
> 		}
> 	}
> }
> options are:
> Options options = new Options();
> 		options.setTo(targetEPR);
> 		options.setUseSeparateListener(false);
> 		options.setAction(wsRequest.getAction());
> 		options.setTimeOutInMilliSeconds(600000);
> 		options.setTransportInProtocol("http");
> 		options.setProperty(org.apache.axis2.context.MessageContextConstants.CHUNKED, org.apache.axis2.transport.http.HTTPConstants.HEADER_TRANSFER_ENCODING);
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message