cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "edison su (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CLOUDSTACK-4477) NPE & Failed to migrate DATA volume from zone wide primary storage 1 to Zone wide primary storage2 on a mixed zone (Xen & VMWARE) after adding Zone
Date Fri, 23 Aug 2013 23:56:54 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

edison su resolved CLOUDSTACK-4477.
-----------------------------------

    Resolution: Fixed
    
> NPE & Failed to migrate DATA volume from zone wide primary storage 1 to Zone wide
primary storage2 on a mixed zone (Xen & VMWARE) after adding Zone 
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4477
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4477
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Storage Controller
>            Reporter: Sailaja Mada
>            Assignee: edison su
>            Priority: Critical
>             Fix For: 4.2.1
>
>         Attachments: migratelogs.rar
>
>
> Steps:
> 1. Configure Adv zone with VMWARE and Xen Clusters in 307 ( Cluster level Primary storage)
> 2. Upgrade to 4.2 
> 3. Add 2nd VMWARE cluster with Cluster level primary storages 
> 4. Add 2 zone wide primary storage's 
> 5. Put cluster1 Primary storage into maintenance 
> 6. Add new DATA volume ( which got deployed on Zone wide primary storage1)
> 7. Attach this volume to an instance(VMWARE)
> 8. Tried to Migrate this volume to Zone wide primary storage2
> Observation:
> NPE while migrating DATA volume from zone wide primary storage 1 to Zone wide primary
storage2 on a mixed zone (Xen & VMWARE) after adding Zone 
> 2013-08-23 20:32:18,063 DEBUG [agent.transport.Request] (StorageManager-Scavenger-1:null)
Seq 7-1010569007: Executing:  { Cmd , MgmtId: 187767034175903, via: 7, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":18,"mountPoint":"/cpg_vol/sailaja/legacyzwps1","path":"357c7fd7962231e99d6482134a65d435","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"c214298f-00cd-32c7-95a9-dd81d8b3dbd1","deviceId":0},"wait":0}}]
}
> 2013-08-23 20:32:18,063 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-376:null)
Seq 7-1010569007: Executing request
> 2013-08-23 20:32:18,070 WARN  [agent.manager.DirectAgentAttache] (DirectAgent-376:null)
Resource returned null answer!
> 2013-08-23 20:32:18,070 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-376:null)
Seq 7-1010569007: Response Received:
> 2013-08-23 20:32:18,071 DEBUG [agent.transport.Request] (DirectAgent-376:null) Seq 7-1010569007:
Processing:  { Ans: , MgmtId: 187767034175903, via: 7, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"Resource
returned null answer","wait":0}}] }
> 2013-08-23 20:32:18,071 DEBUG [agent.manager.AgentAttache] (DirectAgent-376:null) Seq
7-1010569007: No more commands found
> 2013-08-23 20:32:18,071 DEBUG [agent.transport.Request] (StorageManager-Scavenger-1:null)
Seq 7-1010569007: Received:  { Ans: , MgmtId: 187767034175903, via: 7, Ver: v1, Flags: 110,
{ Answer } }
> 2013-08-23 20:32:18,072 INFO  [cloud.template.TemplateManagerImpl] (StorageManager-Scavenger-1:null)
Will retry evicte template: CentOS 5.3(64-bit) no GUI (vSphere) from storage pool: null
> 2013-08-23 20:32:18,098 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null)
Secondary storage garbage collector found 0 volumes to cleanup on volume_store_ref for store:
263abbf4-2997-4dd7-b1ab-9b2abc3a3f2b
> 2013-08-23 20:32:18,116 DEBUG [cache.allocator.StorageCacheRandomAllocator] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Can't find staging storage in zone: 2
> 2013-08-23 20:32:18,546 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy object failed:
> java.lang.NullPointerException
>         at com.cloud.hypervisor.guru.VMwareGuru.resolveNameInGuid(VMwareGuru.java:403)
>         at com.cloud.hypervisor.guru.VMwareGuru.getCommandHostDelegation(VMwareGuru.java:357)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at com.cloud.hypervisor.HypervisorGuruManagerImpl.getGuruProcessedCommandTargetHost(HypervisorGuruManagerImpl.java:64)
>         at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:95)
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
>         at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
>         at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,567 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) copy failed
> com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:231)
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyVolumeBetweenPools(AncientDataMotionStrategy.java:317)
>         at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:406)
>         at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
>         at org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:791)
>  at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2239)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,576 WARN  [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@3ad84031),
no need to delete from object in store ref table
> 2013-08-23 20:32:18,589 WARN  [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@647e4eaf),
no need to delete from object in store ref table
> 2013-08-23 20:32:18,653 DEBUG [agent.transport.Request] (Job-Executor-31:job-137 = [
124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq 13-1361381061: Sending  { Cmd , MgmtId: 187767034175903,
via: 13, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
}
> 2013-08-23 20:32:18,654 DEBUG [agent.transport.Request] (Job-Executor-31:job-137 = [
124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq 13-1361381061: Executing:  { Cmd , MgmtId: 187767034175903,
via: 13, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7262b35e-920e-495e-8a85-7a5c31e5903d","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f528f6ca-31e0-333a-88a1-93acce50e381","id":208,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/legacyzwps2","port":2049}},"name":"sailajadata1","size":5368709120,"path":"bfcdce36b632440d9012c93f16bcea24","volumeId":109,"vmName":"i-4-7-VM","accountId":4,"id":109,"hypervisorType":"VMware"}},"wait":0}}]
}
> 2013-08-23 20:32:18,654 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-462:null)
Seq 13-1361381061: Executing request
> 2013-08-23 20:32:18,803 DEBUG [xen.resource.XenServerStorageProcessor] (DirectAgent-462:null)
Failed to delete volume
> The uuid you supplied was invalid.
>         at com.xensource.xenapi.Types.checkResponse(Types.java:927)
>         at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
>         at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
>         at com.xensource.xenapi.VDI.getByUuid(VDI.java:326)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.deleteVolume(XenServerStorageProcessor.java:452)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:114)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:53)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:617)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
>  at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,839 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-462:null)
Seq 13-1361381061: Response Received:
> 2013-08-23 20:32:18,840 DEBUG [agent.transport.Request] (DirectAgent-462:null) Seq 13-1361381061:
Processing:  { Ans: , MgmtId: 187767034175903, via: 13, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"The
uuid you supplied was invalid.","wait":0}}] }
> 2013-08-23 20:32:18,840 DEBUG [agent.transport.Request] (Job-Executor-31:job-137 = [
124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Seq 13-1361381061: Received:  { Ans: , MgmtId: 187767034175903,
via: 13, Ver: v1, Flags: 10, { Answer } }
> 2013-08-23 20:32:18,850 WARN  [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@fd77e71),
no need to delete from object in store ref table
> 2013-08-23 20:32:18,850 ERROR [cloud.storage.VolumeManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) migrate volume failed:com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
> 2013-08-23 20:32:18,851 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Failed to migrate volume:
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:208] is unreachable:
migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
>         at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2244)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2228)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-23 20:32:18,854 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-137
= [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a ]) Complete async job-137 = [ 124e1b65-d780-486f-b2cb-9cff9f9ff47a
], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to migrate volume
> 2013-08-23 20:32:19,792 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Zone 1 is ready to launch secondary storage VM
> 2013-08-23 20:32:19,815 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
Zone 2 is ready to launch secondary storage VM
> 2013-08-23 20:32:20,100 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null)
Zone 1 is ready to launch console proxy
> 2013-08-23 20:32:20,110 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null)
Zone 2 is ready to launch console proxy

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message