Mailing-List: contact issues-help@cxf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cxf.apache.org
Date: Mon, 9 Jul 2012 16:13:35 +0000 (UTC)
From: "Gary Mulder (JIRA)" <jira@apache.org>
To: issues@cxf.apache.org
Message-ID: <420531752.23687.1341850415147.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Comment Edited] (CXF-2835) ConcurrentModificationException
 in ClientImpl caused by toString-method on requestContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CXF-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409597#comment-13409597 ] 

Gary Mulder edited comment on CXF-2835 at 7/9/12 4:13 PM:
----------------------------------------------------------

We seem to be intermittently reproducing this issue with 2.2.9, so it doesn't seem to be fixed.


Caused by: javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146)
        at $Proxy64.debitFunds(Unknown Source)
        at com.betfair.services.payments.serviceclients.funds.FundsClient.debitFunds(FundsClient.java:76)
        at com.betfair.services.payments.serviceclients.funds.FundsClient$$FastClassByCGLIB$$8a3af175.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
        at com.betfair.monitoring.kpi.aop.KPIMeasuringAspect.measureDuration(KPIMeasuringAspect.java:67)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)

        at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
        at com.betfair.monitoring.kpi.aop.KPIMeasuringAspect.measureDuration(KPIMeasuringAspect.java:67)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        ... 10 more
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$EntryIterator.next(HashMap.java:834)
        at java.util.HashMap$EntryIterator.next(HashMap.java:832)
        at java.util.AbstractMap.toString(AbstractMap.java:485)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at java.util.AbstractMap.toString(AbstractMap.java:490)
        at java.util.Collections$SynchronizedMap.toString(Collections.java:2026)

        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:563)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:514)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        ... 52 more

                
      was (Author: flyingkiwiguy):
    We seem to be intermittently reproducing this issue with 2.2.9, so it doesn't seem to be fixed.


Caused by: javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146)
        at $Proxy64.debitFunds(Unknown Source)
        at com.betfair.services.payments.serviceclients.funds.FundsClient.debitFunds(FundsClient.java:76)
        at com.betfair.services.payments.serviceclients.funds.FundsClient$$FastClassByCGLIB$$8a3af175.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
        at com.betfair.monitoring.kpi.aop.KPIMeasuringAspect.measureDuration(KPIMeasuringAspect.java:67)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)
        at com.betfair.services.payments.serviceclients.funds.FundsClient$$EnhancerByCGLIB$$5c46a949.debitFunds(<generated>)
        at com.betfair.payments.paymentservice.banktransfer.fundstransfer.FundsTransferHandlerImpl.debitFunds(FundsTransferHandlerImpl.java:91)
        at com.betfair.payments.paymentservice.banktransfer.fundstransfer.FundsTransferHandlerImpl.makeWithdrawal(FundsTransferHandlerImpl.java:58)
        at com.betfair.payments.paymentservice.banktransfer.BankTransferHandlerImpl.makeWithdrawal(BankTransferHandlerImpl.java:43)
        at com.betfair.services.payments.dispatcher.BankTransferDispatcher.dispatch(BankTransferDispatcher.java:48)
        at com.betfair.services.payments.ConsolePaymentsFacadeServiceImpl.dispatchPaymentRequest(ConsolePaymentsFacadeServiceImpl.java:144)
        at com.betfair.services.payments.ConsolePaymentsFacadeServiceImpl.processPayment(ConsolePaymentsFacadeServiceImpl.java:96)
        at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
        at com.betfair.monitoring.kpi.aop.KPIMeasuringAspect.measureDuration(KPIMeasuringAspect.java:67)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy73.processPayment(Unknown Source)
        at com.betfair.regpay.services.payments.v1_0.ConsolePaymentsFacadeSyncServiceExecutableResolver$2.execute(ConsolePaymentsFacadeSyncServiceExecutableResolver.java:60)
        at com.betfair.cougar.core.impl.ev.ServiceExecutableResolver$RequestContextExecutable.execute(ServiceExecutableResolver.java:75)
        at com.betfair.cougar.core.impl.ev.InterceptingExecutableWrapper.execute(InterceptingExecutableWrapper.java:43)
        ... 5 more
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$EntryIterator.next(HashMap.java:834)
        at java.util.HashMap$EntryIterator.next(HashMap.java:832)
        at java.util.AbstractMap.toString(AbstractMap.java:485)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at java.util.AbstractMap.toString(AbstractMap.java:490)
        at java.util.Collections$SynchronizedMap.toString(Collections.java:2026)

        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:563)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:514)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
        ... 52 more

                  
> ConcurrentModificationException in ClientImpl caused by toString-method on requestContext
> -----------------------------------------------------------------------------------------
>
>                 Key: CXF-2835
>                 URL: https://issues.apache.org/jira/browse/CXF-2835
>             Project: CXF
>          Issue Type: Bug
>    Affects Versions: 2.2.6
>            Reporter: Henning Jensen
>            Assignee: Daniel Kulp
>             Fix For: 2.2.9
>
>
> We have about 10 threads running at the same time against the same web service. Some of the requests fail with the following stacktrace:
> {code}
> javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set
> 	at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146)
> 	at $Proxy218.getFlightData(Unknown Source)
> 	at com.example.FlightDataAction.run(FlightDataAction.java:94)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> 	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> 	at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> 	at java.util.AbstractMap.toString(AbstractMap.java:485)
> 	at java.lang.String.valueOf(String.java:2826)
> 	at java.lang.StringBuilder.append(StringBuilder.java:115)
> 	at java.util.AbstractMap.toString(AbstractMap.java:490)
> 	at java.util.Collections$SynchronizedMap.toString(Collections.java:2026)
> 	at java.lang.String.valueOf(String.java:2826)
> 	at java.lang.StringBuilder.append(StringBuilder.java:115)
> 	at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:535)
> 	at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:486)
> 	at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:310)
> 	at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:262)
> 	at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
> 	at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
> 	... 5 more
> {code}
> From ClientImpl:
> {code}
>     protected Map<Thread, Map<String, Object>> responseContext 
>         = Collections.synchronizedMap(new WeakHashMap<Thread, Map<String, Object>>());
> ....
>     protected Object[] processResult(Message message,
>                                    Exchange exchange,
>                                    BindingOperationInfo oi,
>                                    Map<String, Object> resContext) throws Exception {
> ....
>         // Grab the response objects if there are any
>         List resList = null;
>         Message inMsg = exchange.getInMessage();
>         if (inMsg != null) {
>             if (null != resContext) {
>                 resContext.putAll(inMsg);
>                 if (LOG.isLoggable(Level.FINE)) {
>                     LOG.fine("set responseContext to be" + responseContext);
>                 }
>             }
>             resList = inMsg.getContent(List.class);
>         }
> ...
> {code}
> The use of toString on responseContext is causing a ConcurrentModificationException when there are many threads working at the same time on the same service. The LOG.fine -line is not optimal since the string is actually resolved no matter if the log line is going to be written to the log file or not. Using String.format() would fix the issue when log level is not FINE, but would not fix the actual problem caused by many threads at the same time. I'm not sure what the developer wish to log in this cause, but limiting the object traversal and not using toString should do the trick. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira