tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muhammad Ali Orakzai <m.orak...@gmail.com>
Subject Re: Web Service Client Response when Old Gen is 100%
Date Thu, 07 Nov 2013 04:23:30 GMT
I am using GUI version of tomcat and heap memory has been configured to
500M min and 1000M max. So memory size is not an issue. GC is being
performed normally. Took a heap dump on 60% memory. Analyzing it in Eclipse
MAT showed the following leak suspect.

One instance of *"java.util.Vector"* loaded by *"<system class
loader>"*occupies *335,247,200
(70.86%)* bytes. The instance is referenced by
*org.apache.tomcat.util.threads.TaskThread
@ 0x1ec45d98 http-apr-8082-exec-11* , loaded by
*"org.apache.catalina.loader.StandardClassLoader
@ 0x176e0ef0"*. The memory is accumulated in one instance of
*"java.lang.Object[]"* loaded by *"<system class loader>"*.


This was the stack trace.


http-apr-8082-exec-11
  at java.util.Vector$1.nextElement()Ljava/lang/Object; (Unknown Source)
  at com.sun.org.apache.xml.internal.resolver.Catalog.resolveLocalSystem(Ljava/lang/String;)Ljava/lang/String;
(Unknown Source)
  at com.sun.org.apache.xml.internal.resolver.Catalog.resolveSystem(Ljava/lang/String;)Ljava/lang/String;
(Unknown Source)
  at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.getResolvedEntity(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
(Unknown Source)
  at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lorg/xml/sax/InputSource;
(Unknown Source)
  at com.sun.xml.ws.wsdl.parser.EntityResolverWrapper.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser;
(EntityResolverWrapper.java:63)
  at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser;
(RuntimeWSDLParser.java:287)
  at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;Lcom/sun/xml/ws/api/policy/PolicyResolver;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl;
(RuntimeWSDLParser.java:151)
  at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl;
(RuntimeWSDLParser.java:131)
  at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl;
(WSServiceDelegate.java:267)
  at com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljavax/xml/transform/Source;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V
(WSServiceDelegate.java:230)
  at com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V
(WSServiceDelegate.java:178)
  at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)Ljavax/xml/ws/spi/ServiceDelegate;
(ProviderImpl.java:106)
  at javax.xml.ws.Service.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;)V
(Unknown Source)
  at com.example.xmlns._1366097065072.CashRechargeServiceServiceagent.<init>()V
(CashRechargeServiceServiceagent.java:42)
  at pk.com.telenor.topup.cbs.CbsCashRecharge.cashRecharge(Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeInput;)Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeOutput;
(CbsCashRecharge.java:119)
  at pk.com.telenor.topup.cbs.CbsCashRecharge.performCardRechargePerform(Lpk/com/telenor/topup/dto/CashRechargeRequest;)Lpk/com/telenor/topup/dto/CashRechargeResponse;
(CbsCashRecharge.java:58)
  at telenor.vas.topup.TopUpWS.TopUp(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ltelenor/vas/topup/PaymentResponse;
(TopUpWS.java:409)
  at sun.reflect.GeneratedMethodAccessor129.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
  at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
  at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;
(InstanceResolver.java:246)
  at com.sun.xml.ws.server.InvokerTube$2.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;
(InvokerTube.java:146)
  at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet;
(EndpointMethodHandler.java:257)
  at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/pipe/NextAction;
(SEIInvokerTube.java:95)
  at com.sun.xml.ws.api.pipe.Fiber.__doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube;
(Fiber.java:629)
  at com.sun.xml.ws.api.pipe.Fiber._doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube;
(Fiber.java:588)
  at com.sun.xml.ws.api.pipe.Fiber.doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube;
(Fiber.java:573)
  at com.sun.xml.ws.api.pipe.Fiber.runSync(Lcom/sun/xml/ws/api/pipe/Tube;Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet;
(Fiber.java:470)
  at com.sun.xml.ws.server.WSEndpointImpl$2.process(Lcom/sun/xml/ws/api/message/Packet;Lcom/sun/xml/ws/api/server/WebServiceContextDelegate;Lcom/sun/xml/ws/api/server/TransportBackChannel;)Lcom/sun/xml/ws/api/message/Packet;
(WSEndpointImpl.java:295)
  at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V
(HttpAdapter.java:515)
  at com.sun.xml.ws.transport.http.HttpAdapter.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V
(HttpAdapter.java:285)
  at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(Ljavax/servlet/ServletContext;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
(ServletAdapter.java:143)
  at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V
(WSServletDelegate.java:155)
  at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V
(WSServletDelegate.java:189)
  at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
(WSServlet.java:76)
  at javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
(HttpServlet.java:647)
  at javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(HttpServlet.java:728)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(ApplicationFilterChain.java:305)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
(ApplicationFilterChain.java:210)
  at org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(StandardWrapperValve.java:222)
  at org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(StandardContextValve.java:123)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(AuthenticatorBase.java:472)
  at org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(StandardHostValve.java:171)
  at org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(ErrorReportValve.java:99)
  at org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(AccessLogValve.java:936)
  at org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
(StandardEngineValve.java:118)
  at org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V
(CoyoteAdapter.java:407)
  at org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
(AbstractHttp11Processor.java:1004)
  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
(AbstractProtocol.java:589)
  at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()V
(AprEndpoint.java:1852)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (Unknown Source)
  at java.lang.Thread.run()V (Unknown Source)


It is stuck in the web service call. My web service client is embedded
in a jar file. Jar file is being used in 5 web apps deployed on this
server. Strange thing is that similar services calls have been used
and they are working fine without any delays. Delay is always in this
web service call.



On Wed, Nov 6, 2013 at 8:25 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Muhammad,
>
> On 11/6/13, 1:43 AM, Muhammad Ali Orakzai wrote:
> > -XX:MaxPermSize=100m
>
> You don't have any other heap-sizing parameters. Perhaps you simply
> need more memory? It used to be that the default heap sizes were very
> small (sometimes like 64MiB) and you may simply need much more. How
> much memory is the JVM actually running with given your existing
> command-line? You can check with "jinfo" from the command-line.
>
> > -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
> > -XX:+HeapDumpOnOutOfMemoryError
>
> Have you inspected one of these? You should get one for each OOM. You
> should be able to find out what kinds of objects are filling-up your
> heap. It should give you some clue as to what kind of action you
> should take next: either raise your heap size or fix your web application.
>
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.port=9004
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
>
> Um... the above doesn't look like a good idea, but only you know your
> own environment.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.15 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJSel9OAAoJEBzwKT+lPKRYmDQP/3ZcHDmotCo1ZdX2IKMlfHAZ
> kHHZuuU62uAXhMDABYn768M0jKqs90P5yBPcb09PYxB0ZuPBSgq5hgANWAD/mcIc
> l558lg5LAc5yfbZedgLZzuMA+13/G/z8jBnND1j1S0psaMYA8LVk3ExidxwXbBB+
> WEpRCZNhMovZkL2rHxO37vYGP0DNUtREjpVs+S4fWOJCYRrQmajfBwMttTVl36U/
> 2md195/N0JccdCa119kV8WACIr5jeDXa6al3raP+dC893SUW2pZB/M6Wxk6eLND0
> q8bIRjGuzfqZNn+tupNKKJqakT5kW8xASMrjUKKTq0/l6+nHD3DDSMa8Qi0YyWFD
> VL3YuW7IYCpGjniCWrXAeB5KxueS7OUoSq5EM290fs51ZYSY/LF5Stb7ybEAAL9a
> e4oisal9u/Jc3ZRsH3hgJDJusgAL7lF1GCOW/DVf5pvbrL75SdslAO67j92fJ3GI
> 9YtBQucpYLNvfUyqf+fOde5bcdECImmo2nM7fOt6fmd9rQBPNLC+holwvbixqfZO
> /Ny6e6NrRfxvHHw45/T0xE0IjUufSpmQQ0/n5hISg2YUtoEfIUqvc3Xv7FCEa+H4
> hlDapQmJPGhfPWZhJSN8DqiGVHMBf/dUcyHfCeIb4Hs1FczX7/EXel6Pwn86N2MZ
> 3LDowe2VrWhevlfDBL07
> =bzZV
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message