cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Likitha Shetty (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-7652) Various operations in VMware setup intermittently fail with 'javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or has not been completely created'
Date Tue, 30 Sep 2014 08:27:34 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Likitha Shetty updated CLOUDSTACK-7652:
---------------------------------------
    Summary: Various operations in VMware setup intermittently fail with 'javax.xml.ws.soap.SOAPFaultException:
The object has already been deleted or has not been completely created'  (was: Various operation
in VMware setup intermittently fail with 'javax.xml.ws.soap.SOAPFaultException: The object
has already been deleted or has not been completely created')

> Various operations in VMware setup intermittently fail with 'javax.xml.ws.soap.SOAPFaultException:
The object has already been deleted or has not been completely created'
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-7652
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7652
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: VMware
>    Affects Versions: 4.5.0
>            Reporter: Likitha Shetty
>            Assignee: Likitha Shetty
>            Priority: Critical
>             Fix For: 4.5.0
>
>
> +Steps to reproduce+ 
> 1. Setup VMware 4.5 with 1 CCP Management server.
> 2. Create 2 VMware DCs in 2 different vCenters.
> 3. Add 2 zones in CCP corresponding to the two DCs.
> 4. Do basic VM/Volume lifecycle operations.
> Keep monitoring the logs for the following error - *javax.xml.ws.soap.SOAPFaultException:
The object has already been deleted or has not been completely created*
> For e.g. 
> {noformat}
> javax.xml.ws.soap.SOAPFaultException
> Message: The object has already been deleted or has not been completely created
> javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or has not
been completely created
>         at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
>         at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:119)
>         at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108)
>         at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
>         at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:110)
>         at com.sun.proxy.$Proxy389.retrieveProperties(Unknown Source)
>         at com.cloud.hypervisor.vmware.util.VmwareClient.retrieveMoRefProperties(VmwareClient.java:308)
>         at com.cloud.hypervisor.vmware.util.VmwareClient.getDynamicProperty(VmwareClient.java:263)
>         at com.cloud.hypervisor.vmware.mo.HostMO.getHostDatastoreSystemMO(HostMO.java:166)
>         at com.cloud.hypervisor.vmware.mo.HostMO.findDatastore(HostMO.java:889)
>         at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.findDatastoreWithBackwardsCompatibility(HypervisorHostHelper.java:131)
>         at com.cloud.storage.resource.VmwareStorageProcessor.deleteVolume(VmwareStorageProcessor.java:1481)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:120)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:54)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:598)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:215)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:701)
> {noformat}
> +Root Cause Analysis+
> The thread local context which is used to handle connections to a vCenter is being re-used
intermittently. And when it is re-used, if the existing context corresponds to the other vCenter
then the operation being executed will fail.
> The re-use is happening because after the PingTask completion we are not clearing the
ThreadLocal context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message