chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bindu Wavell <bwav...@google.com>
Subject Re: Infinite loop in AtomEntryParser
Date Wed, 19 Mar 2014 21:18:01 GMT
I believe they are using 3.2.4, assuming this is the right
jar: wstx-asl-3.2.4.jar


On Wed, Mar 19, 2014 at 2:11 PM, Florian Müller <fmui@apache.org> wrote:

> Hi Bindu,
>
> I've never seen something like this. Do you know which Woodstox version
> is used?
>
> The CappedInputStream is something different. It prevents malicious
> clients from sending an endless XML and consuming server resources.
>
>
> - Florian
>
>
>
> > Our server is Alfresco Enterprise 4.1.3.0 which is in turn using
> > openchemistry 0.7.0. Our client is Alfresco Workdesk 4.1.1.1 which is in
> > turn using openchemistry 0.8.0.
> >
> > For the last few weeks we have run into a situation where the Alfresco
> > server becomes unresponsive. Specifically it doesn't crash it simply
> > becomes so overloaded that we can't login via Workdesk/CMIS. Using
> > JConsole and subsequently YourKit we can see that the server runs fine
> > and uses between 3-6GB of RAM with nice slow growth and cleanup. Then at
> > some point we see a huge amount of memory allocation and deallocation
> > between 3GB and 15GB every couple of seconds and this occurs forever
> > until we restart the server. The parallel GC cleaning up this memory
> > consumes so much CPU that folks can't actually use the system.
> >
> > To begin with we took a stackshot every 10 seconds for a minute. We saw
> > that there were 5 opencmis threads "frozen" while parsing an AtomPub
> > document. They were not deadlocked. Subsequently we found the same thing
> > using YourKit, it calls these frozen threads. Following is an example
> > stack trace from YourKit:
> >
> > http-127.0.0.1-8080-2 <--- Frozen for at least 47m 15 sec
> > com.ctc.wstx.sr.StreamScanner.getNext()
> > com.ctc.wstx.sr.BasicStreamReader.skipToken()
> > com.ctc.wstx.sr.BasicStreamReader.nextFromTree()
> > com.ctc.wstx.sr.BasicStreamReader.next()
> >
> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.next(XMLStreamReader)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.skip(XMLStreamReader)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.parseCmisContent(XMLStreamReader)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.parseEntry(XMLStreamReader)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.parse(InputStream)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.ObjectService.create(CallContext,
> > CmisService, String, HttpServletRequest, HttpServletResponse)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Method, Object, Object[])
> > sun.reflect.NativeMethodAccessorImpl.invoke(Object, Object[])
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[])
> > java.lang.reflect.Method.invoke(Object, Object[])
> > org.apache.chemistry.opencmis.server.shared.Dispatcher.dispatch(String,
> > String, CallContext, CmisService, String, HttpServletRequest,
> > HttpServletResponse)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.dispatch(CallContext,
> > HttpServletRequest, HttpServletResponse)
> >
> org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.service(HttpServletRequest,
> > HttpServletResponse)
> > javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
> > ServletResponse)
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
> > ServletResponse)
> >
> org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(ServletRequest,
> > ServletResponse, FilterChain)
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
> > ServletResponse)
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
> > ServletResponse)
> > org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> > org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Request,
> > Response)
> > org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> > org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> > org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> > org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> > org.apache.coyote.http11.Http11Processor.process(Socket)
> >
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Socket)
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run()
> > java.lang.Thread.run()
> >
> > By inspecting garbage collected objects we can see that this is almost
> > all string[] tied back to AtomEntryParser (and the woodstox parser stuff
> > under it.) If we look at one of these arrays we see the same stuff over
> > and over again... see a screenshot here of the first few elements in
> > such an array: http://i.imgur.com/RzNQBYG.png . There are a bunch of
> > these VERY large string arrays and they all look like they are looping
> > over the front of an AtomPub document.
> >
> > I noticed in opencmis 0.9.0 in the implementation of AtomEntryParser
> > that a CappedInputStream was added while parsing the AtomPub stream.
> > This appears to be limited to 10MB. Not sure if that is related to this
> > issue. Or if others have run into this issue. Any thought/guidance would
> > be greatly appreciated.
> >
> > I'm bcc'ing Florian and Gab (the spy) but figured it was worth having
> > this public as I've been trying to track this issue down for a while and
> > have not found an answer.
> >
> > Thanks all!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message