Mailing-List: contact issues-help@cxf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cxf.apache.org
Message-ID: <18271469.51571291948621358.JavaMail.jira@thor>
Date: Thu, 9 Dec 2010 21:37:01 -0500 (EST)
From: "Willem Jiang (JIRA)" <jira@apache.org>
To: issues@cxf.apache.org
Subject: [jira] Created: (CXF-3180) Jaxrs/Jaxb unmarshaller code regularly
 causing performance problems & thread lock-up
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Jaxrs/Jaxb unmarshaller code regularly causing performance problems & thread lock-up
------------------------------------------------------------------------------------

                 Key: CXF-3180
                 URL: https://issues.apache.org/jira/browse/CXF-3180
             Project: CXF
          Issue Type: Bug
    Affects Versions: 2.3.1, 2.2.12, 2.2.11, 2.3.0, 2.2.10, 2.2.9, 2.2.8, 2.2.7, 2.2.6
            Reporter: Willem Jiang
            Assignee: Willem Jiang
             Fix For: 2.3.2, 2.4


Please find below the details of the XML parsing performance bottleneck we are seeing when processing REST POST requests with XML bodies.
The performance issue is caused by the fact that CXF code creates a new JAXB Unmarshaller for every incoming request. The Unmarshaller is created quickly enough but when it is used to do the unmarshalling, it needs to create a Xerces SAXParser, which in turn calls DTDDVFactory.getInstance() and that method is synchronized. When we try to process more than 200-300 requests per second we start seeing a lot of threads blocked in DTDDVFactory.getInstance().

The thread stack of one of the blocked threads is:

1390532156@qtp-69728034-390" prio=10 tid=0x00002aab449c0000 nid=0x28f4 waiting for monitor entry [0x000000004da26000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)

waiting to lock <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.newInstance(ObjectFactory.java:349)
at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:154)
at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:97)
at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:102)
at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:87)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.<init>(SAXParserImpl.java:332)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:122)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:76)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:75)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:184)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:191)
at org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:216)
at org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:186)
at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:157)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:947)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:560)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:525)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:230)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:88)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
locked <0x00002aab1e41a0e8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:109)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:312)
...
The application where the problem is seen defines an XSD file and uses theJAXB compiler to create the data binding java class. It then creates a "RESTResouce" java class that is CXF/JAXRS annotated and contains a method that defines the processing of incoming POST requests. That method has one argument, the JAXB-compiled class, which tells CXF to unmarshall the XML body of the incoming request.

Just in case, you need more context, here is the top portion of the stack of the thread that is inside the synchronized method:

"1840040935@qtp-69728034-391" prio=10 tid=0x00002aab44810000 nid=0x2dfa waiting on condition [0x000000005be59000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:211)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:381)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:457)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:398)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:105)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.eclipse.osgi.internal.loader.BundleLoader.loadClass(BundleLoader.java:326)
at org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:231)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1193)
at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)
at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.findProviderClass(ObjectFactory.java:395)
at com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.newInstance(ObjectFactory.java:350)
at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:59)

locked <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)
locked <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
...


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.