cxf-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Salt (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CXF-7135) ConcurrentModificationException in MessageImpl.calcContextCache() when using JMS Transport and JAXRS Client
Date Wed, 16 Nov 2016 14:55:58 GMT

     [ https://issues.apache.org/jira/browse/CXF-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dan Salt updated CXF-7135:
--------------------------
    Description: 
Our platform allows the user to switch transports for a particular proxy based on configuration.
We currently allow HTTP and JMS as possible options.

During recent performance testing, we've hit a bug which causes JAXRS Clients over JMS to
fail with a ConcurrentModificationException:

A test case is attached that reproduces our problem.

Our test executes requests across multiple concurrent threads, switching between JMS and HTTP
for both JAXWS and JAXRS Client Proxies. In the test case:

- JAXWS over both HTTP and JMS works fine, both in single and multi-threaded tests.
- JAXRS works fine across single and multi-threaded tests over HTTP with the 'ThreadSafe'
flag set on the Client Factory. 
- JAXRS over JMS fails in both single-threaded and multi-threaded mode with error (1) below.
In multi-threaded, also fails with (2) below. 

1) The ConnectionFactoryFeature fails to properly set the JMS Connection Factory on the Conduit
and results in an error at send-time because no Connection Factory is set. This is because
in JAXRS Clients, the InterceptorProvider is neither a Client or Server, it is a ClientConfiguration,
so the JMS ConnectionFactoryFeature does not fire either of it's methods:

public void initialize(Client client, Bus bus) 
public void initialize(Server server, Bus bus)

We have worked around this in our code by sub-classing and overriding the method:
public void initialize(InterceptorProvider interceptorProvider, Bus bus)

.. and adding code similar to that in the ConnectionFactory feature. This resolves the missing
ConnectionFactory problem and JAXRS-over-JMS works fine in single-threaded mode.

2) Under multi-threaded conditions, the JAXRS Clients fail with the following exception:

javax.ws.rs.client.ResponseProcessingException: Problem with reading the data, class java.lang.String,
ContentType: text/plain.
	at org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
	at org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
	at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
	at com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
	at java.util.HashMap.putMapEntries(HashMap.java:511)
	at java.util.HashMap.putAll(HashMap.java:784)
	at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
	at org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
	at org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
	at org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
	at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
	... 7 more

This occurs even though the 'ThreadSafe' property is set via:

rsClientFactory.setThreadSafe(true);

Also, interestingly, this exception occurs even if I put the entire Proxy Client invocation
in a synchronized block (which should effectively render the Clients single-threaded). 

Finally, one important factor is that this error only happens on the latest 3.0.x release
(3.0.11). When I switch my test case to 3.1.8, it does NOT occur. Not sure if this equates
to a fix not being back-ported, or a fundamental difference in functionality. Sadly, we're
not able to yet move to 3.1.x, because of running inside the Karaf container, and our dependencies
on Camel and Spring 3.x.

I've done quite a large amount of digging through the code to try and narrow down the scope
of the problem, and the results above is as far as I could get, unfortunately. My guess/assumption
is that there are some JMS-specific objects that are being added to the Request/Response context
which are being added/removed in a non-thread safe way. Purely a guess, though.

Thanks. Please let me know if you need any other information.

  was:
Our platform allows the user to switch transports for a particular proxy based on configuration.
We currently allow HTTP and JMS as possible options.

During recent performance testing, we've hit a bug which causes JAXRS Clients over JMS to
fail with a ConcurrentModificationException:

A test case is attached that reproduces our problem.

Our test executes requests across multiple concurrent threads, switching between JMS and HTTP
for both JAXWS and JAXRS Client Proxies. In the test case:

- JAXWS over both HTTP and JMS works fine, both in single and multi-threaded tests.
- JAXRS works fine across single and multi-threaded tests over HTTP with the 'ThreadSafe'
flag set on the Client Factory. 
- JAXRS over JMS fails in both single-threaded and multi-threaded mode with error (1) below.
In multi-threaded, also fails with (2) below. 

1) The ConnectionFactoryFeature fails to properly set the JMS Connection Factory on the Conduit
and results in an error at send-time because no Connection Factory is set. This is because
in JAXRS Clients, the InterceptorProvider is neither a Client or Server, it is a ClientConfiguration,
so the JMS ConnectionFactoryFeature does not fire either of it's methods:

public void initialize(Client client, Bus bus) 
public void initialize(Server server, Bus bus)

We have worked around this in our code by sub-classing and overriding the method:
public void initialize(InterceptorProvider interceptorProvider, Bus bus)

.. and adding code similar to that in the ConnectionFactory feature. This resolves the missing
ConnectionFactory problem and JAXRS-over-JMS works fine in single-threaded mode.

2) Under multi-threaded conditions, the JAXRS Clients fail with the following exception:

javax.ws.rs.client.ResponseProcessingException: Problem with reading the data, class java.lang.String,
ContentType: text/plain.
	at org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
	at org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
	at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
	at com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
	at java.util.HashMap.putMapEntries(HashMap.java:511)
	at java.util.HashMap.putAll(HashMap.java:784)
	at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
	at org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
	at org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
	at org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
	at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
	... 7 more

This occurs even though the 'ThreadSafe' property is set via:

rsClientFactory.setThreadSafe(true);

I've done quite a large amount of digging through the code to try and narrow down the scope
of the problem, and the results above is as far as I could get, unfortunately. My guess/assumption
is that there are some JMS-specific objects that are being added to the Request/Response context
which are being added/removed in a non-thread safe way. Purely a guess, though.

Finally, one important factor is that this error only happens on the latest 3.0.x release
(3.0.11). When I switch my test case to 3.1.8, it does NOT occur. Not sure if this equates
to a fix not being back-ported, or a fundamental difference in functionality. Sadly, we're
not able to yet move to 3.1.x, because of running inside the Karaf container, and our dependencies
on Camel and Spring 3.x.

Thanks. Please let me know if you need any other information.


> ConcurrentModificationException in MessageImpl.calcContextCache() when using JMS Transport
and JAXRS Client
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: CXF-7135
>                 URL: https://issues.apache.org/jira/browse/CXF-7135
>             Project: CXF
>          Issue Type: Bug
>          Components: JAX-RS, JMS
>    Affects Versions: 3.0.11
>         Environment: Mac OSX 10.11.6. 
> Java JDK 1.8.0_71 (64 bit Server)
>            Reporter: Dan Salt
>         Attachments: cxf-rsjms-testcase.zip
>
>
> Our platform allows the user to switch transports for a particular proxy based on configuration.
We currently allow HTTP and JMS as possible options.
> During recent performance testing, we've hit a bug which causes JAXRS Clients over JMS
to fail with a ConcurrentModificationException:
> A test case is attached that reproduces our problem.
> Our test executes requests across multiple concurrent threads, switching between JMS
and HTTP for both JAXWS and JAXRS Client Proxies. In the test case:
> - JAXWS over both HTTP and JMS works fine, both in single and multi-threaded tests.
> - JAXRS works fine across single and multi-threaded tests over HTTP with the 'ThreadSafe'
flag set on the Client Factory. 
> - JAXRS over JMS fails in both single-threaded and multi-threaded mode with error (1)
below. In multi-threaded, also fails with (2) below. 
> 1) The ConnectionFactoryFeature fails to properly set the JMS Connection Factory on the
Conduit and results in an error at send-time because no Connection Factory is set. This is
because in JAXRS Clients, the InterceptorProvider is neither a Client or Server, it is a ClientConfiguration,
so the JMS ConnectionFactoryFeature does not fire either of it's methods:
> public void initialize(Client client, Bus bus) 
> public void initialize(Server server, Bus bus)
> We have worked around this in our code by sub-classing and overriding the method:
> public void initialize(InterceptorProvider interceptorProvider, Bus bus)
> .. and adding code similar to that in the ConnectionFactory feature. This resolves the
missing ConnectionFactory problem and JAXRS-over-JMS works fine in single-threaded mode.
> 2) Under multi-threaded conditions, the JAXRS Clients fail with the following exception:
> javax.ws.rs.client.ResponseProcessingException: Problem with reading the data, class
java.lang.String, ContentType: text/plain.
> 	at org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
> 	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
> 	at org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
> 	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
> 	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
> 	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
> 	at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
> 	at com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> 	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
> 	at java.util.HashMap.putMapEntries(HashMap.java:511)
> 	at java.util.HashMap.putAll(HashMap.java:784)
> 	at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
> 	at org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
> 	at org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
> 	at org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
> 	at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
> 	at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
> 	... 7 more
> This occurs even though the 'ThreadSafe' property is set via:
> rsClientFactory.setThreadSafe(true);
> Also, interestingly, this exception occurs even if I put the entire Proxy Client invocation
in a synchronized block (which should effectively render the Clients single-threaded). 
> Finally, one important factor is that this error only happens on the latest 3.0.x release
(3.0.11). When I switch my test case to 3.1.8, it does NOT occur. Not sure if this equates
to a fix not being back-ported, or a fundamental difference in functionality. Sadly, we're
not able to yet move to 3.1.x, because of running inside the Karaf container, and our dependencies
on Camel and Spring 3.x.
> I've done quite a large amount of digging through the code to try and narrow down the
scope of the problem, and the results above is as far as I could get, unfortunately. My guess/assumption
is that there are some JMS-specific objects that are being added to the Request/Response context
which are being added/removed in a non-thread safe way. Purely a guess, though.
> Thanks. Please let me know if you need any other information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message