Mailing-List: contact issues-help@cxf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cxf.apache.org
Date: Thu, 12 Nov 2015 11:49:11 +0000 (UTC)
From: "Marc Layer (JIRA)" <jira@apache.org>
To: issues@cxf.apache.org
Message-ID: <JIRA.12912455.1447328756000.49316.1447328951355@Atlassian.JIRA>
In-Reply-To: <JIRA.12912455.1447328756000@Atlassian.JIRA>
References: <JIRA.12912455.1447328756000@Atlassian.JIRA>
 <JIRA.12912455.1447328756462@arcas>
Subject: [jira] [Updated] (CXF-6674) Infinite loop when handling SOAP
 responses in split/parallel Camel route
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CXF-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marc Layer updated CXF-6674:
----------------------------
    Description: 
We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some backend SOAP web services. In our production environment we are facing the issue that at irregular intervals a CXF worker thread would hang in a busy loop, consuming 100% CPU time. The only recovery so far is to restart the server.

The loop seems to occur in {{org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped()}}:
{noformat}
	private void copyScoped(Map<String, Object> msg) {
		for (String s : scopes.keySet()) {
			message.put(s, msg.get(s));
		}
	}
{noformat}

I guess it might be related to a threading issue that causes the iterator of scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I can see that the iterated "scope" map is an unsynchronized HashMap. I am unsure where the "scope" map was created and how it could have leaked into another thread. "message" seems to be an instance of MessageImpl, which is also derived from HashMap and therefore unsynchronized. I attached a Dynatrace screenshot from one such invocation.

Multi-threading in our case is two-fold:
- CXF seems to be using separate worker threads even for synchronous invocations.
- Additionally the call in question makes use of Camel's splitter EIP with parallel processing which also uses separate threads (RouteBuilder is attached), effectively like this:
{noformat}
	.split(body(), dataAggregator)
		.stopOnException()
		.parallelProcessing().executorServiceRef("multicastExecutorService")
		.to("cxf:bean:backendService")
{noformat}

I analyzed a number of cases in which two web service calls were dispatched at the same time. Dynatrace shows that the backend web services always finished within about 80-130ms.

Unfortunately we were unable to reproduce the error on a testing system where we could have attached a remote debugger to the application, not even under heavy load. It only occured in the production environment so far. Therefore I am uncertain if the problem only occurs when Camel's parallelProcessing() is used. We will disable this feature in the application's next release (end of next week) to verify this.


There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of ConcurrentHashMap rather than a subclass of StringImpl (which is also derived from HashMap). Maybe these issues are related. The fix, however, was implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as Websphere 8 is only available for IBM JRE 1.6. The latest Camel version available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only available for Java 1.7. I think we could easily upgrade to a newer 2.7.x version of CXF but not to CXF 3.x.


  was:
We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some backend SOAP web services. In our production environment we are facing the issue that at irregular intervals a CXF worker thread would hang in a busy loop, consuming 100% CPU time. The only recovery so far is to restart the server.

The loop seems to occur in org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped():

	private void copyScoped(Map<String, Object> msg) {
		for (String s : scopes.keySet()) {
			message.put(s, msg.get(s));
		}
	}

I guess it might be related to a threading issue that causes the iterator of scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I can see that the iterated "scope" map is an unsynchronized HashMap. I am unsure where the "scope" map was created and how it could have leaked into another thread. "message" seems to be an instance of MessageImpl, which is also derived from HashMap and therefore unsynchronized. I attached a Dynatrace screenshot from one such invocation.

Multi-threading in our case is two-fold:
- CXF seems to be using separate worker threads even for synchronous invocations.
- Additionally the call in question makes use of Camel's splitter EIP with parallel processing which also uses separate threads (RouteBuilder is attached), effectively like this:

	.split(body(), dataAggregator)
		.stopOnException()
		.parallelProcessing().executorServiceRef("multicastExecutorService")
		.to("cxf:bean:backendService")

I analyzed a number of cases in which two web service calls were dispatched at the same time. Dynatrace shows that the backend web services always finished within about 80-130ms.

Unfortunately we were unable to reproduce the error on a testing system where we could have attached a remote debugger to the application, not even under heavy load. It only occured in the production environment so far. Therefore I am uncertain if the problem only occurs when Camel's parallelProcessing() is used. We will disable this feature in the application's next release (end of next week) to verify this.


There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of ConcurrentHashMap rather than a subclass of StringImpl (which is also derived from HashMap). Maybe these issues are related. The fix, however, was implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as Websphere 8 is only available for IBM JRE 1.6. The latest Camel version available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only available for Java 1.7. I think we could easily upgrade to a newer 2.7.x version of CXF but not to CXF 3.x.


> Infinite loop when handling SOAP responses in split/parallel Camel route
> ------------------------------------------------------------------------
>
>                 Key: CXF-6674
>                 URL: https://issues.apache.org/jira/browse/CXF-6674
>             Project: CXF
>          Issue Type: Bug
>          Components: JAX-WS Runtime
>    Affects Versions: 2.7.10
>         Environment: Apache Camel
> 2.12.3
> Apache CXF
> 2.7.10
> App Server
> WebSphere Platform 8.0.0.10 [ND 8.0.0.10 cf101502.03]
> Java Runtime Version
> pxa6460_26sr8fp2ifx-20141114_01 (SR8 FP2)
> Java Vendor
> IBM Corporation
> Java Version
> 1.6.0
> Java Compiler
> j9jit26
> Java VM name
> IBM J9 VM
> Operating System
> Linux
> OS Architecture
> x86_64
> OS Version
> 2.6.32-279.el6.x86_64
> VM Information
> JRE 1.6.0 Linux amd64-64 Compressed References 20141114_222464 (JIT enabled, AOT enabled)
> J9VM - R26_Java626_SR8_20141114_1023_B222464
> JIT  - r11.b07_20141003_74578.01
> GC   - R26_Java626_SR8_20141114_1023_B222464_CMPRSS
> J9CL - 20141114_222464
> VM Vendor
> IBM Corporation
> VM Version
> 2.6
>            Reporter: Marc Layer
>         Attachments: BackendServiceRoute.java, dynatrace.png
>
>
> We are using Apache Camel 2.12.3 and Apache CXF 2.7.10 to integrate some backend SOAP web services. In our production environment we are facing the issue that at irregular intervals a CXF worker thread would hang in a busy loop, consuming 100% CPU time. The only recovery so far is to restart the server.
> The loop seems to occur in {{org.apache.cxf.jaxws.context.WrappedMessageContext.copyScoped()}}:
> {noformat}
> 	private void copyScoped(Map<String, Object> msg) {
> 		for (String s : scopes.keySet()) {
> 			message.put(s, msg.get(s));
> 		}
> 	}
> {noformat}
> I guess it might be related to a threading issue that causes the iterator of scopes.keySet() to loop infinitely. Using the VM monitoring tool Dynatrace, I can see that the iterated "scope" map is an unsynchronized HashMap. I am unsure where the "scope" map was created and how it could have leaked into another thread. "message" seems to be an instance of MessageImpl, which is also derived from HashMap and therefore unsynchronized. I attached a Dynatrace screenshot from one such invocation.
> Multi-threading in our case is two-fold:
> - CXF seems to be using separate worker threads even for synchronous invocations.
> - Additionally the call in question makes use of Camel's splitter EIP with parallel processing which also uses separate threads (RouteBuilder is attached), effectively like this:
> {noformat}
> 	.split(body(), dataAggregator)
> 		.stopOnException()
> 		.parallelProcessing().executorServiceRef("multicastExecutorService")
> 		.to("cxf:bean:backendService")
> {noformat}
> I analyzed a number of cases in which two web service calls were dispatched at the same time. Dynatrace shows that the backend web services always finished within about 80-130ms.
> Unfortunately we were unable to reproduce the error on a testing system where we could have attached a remote debugger to the application, not even under heavy load. It only occured in the production environment so far. Therefore I am uncertain if the problem only occurs when Camel's parallelProcessing() is used. We will disable this feature in the application's next release (end of next week) to verify this.
> There was another issue, CXF-6160 "AsyncResponse infinite loops", in which an infinite loop in MessageImpl was tackled by making ExchangeImpl a subclass of ConcurrentHashMap rather than a subclass of StringImpl (which is also derived from HashMap). Maybe these issues are related. The fix, however, was implemented only for CXF 3.x. Unfortunately we are bound to using Java 1.6 as Websphere 8 is only available for IBM JRE 1.6. The latest Camel version available for Java 1.6 comes with CXF 2.7.14, later Camel versions are only available for Java 1.7. I think we could easily upgrade to a newer 2.7.x version of CXF but not to CXF 3.x.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)