cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailaja Mada (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-4228) [VMWARE]NPE & operation timed out issues while migrating volumes across different Zone wide primary storages
Date Sat, 10 Aug 2013 08:43:54 GMT
Sailaja Mada created CLOUDSTACK-4228:
----------------------------------------

             Summary: [VMWARE]NPE & operation timed out issues while migrating volumes
across different Zone wide primary storages
                 Key: CLOUDSTACK-4228
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Storage Controller, VMware
    Affects Versions: 4.2.0
            Reporter: Sailaja Mada
            Priority: Critical


Steps:

1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster 
2. Configured new zone with VMWARE cluster of 2 hosts  with zone wide primary storage 
3. Added one more Zone wide primary storage to this new zone
4. Deploy instance on this zone 
5. Add new DATA volume and attach to this instance 
6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary storage 1 to
second Zone wide primary storage which is added to the same zone 

Observation:
1. NPE & operation timed out issues are observed while migrating volumes across different
Zone wide primary storage's

2. It is observed 3 times when tried about 6 times. 

log & DB dumb is attached . This is a single node management server 

2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Waiting some more time because this is the current command
2013-08-10 00:02:25,453 INFO  [utils.exception.CSExceptionErrorCode] (StorageManager-Scavenger-4:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2013-08-10 00:02:25,453 WARN  [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Timed out on Seq 18-1413808894:  { Cmd , MgmtId: 90310994128556, via: 18,
Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Cancelling.
2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: No more commands found
2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-4:null)
Unable to send storage pool command to org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2
via 18
com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host 18 timed out after
3600
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439)
        at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987)
        at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438)
        at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452)
        at com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793)
        at com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039)
        at com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null)
Seq 14-647496884: Sending  { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null)
Seq 14-647496884: Executing:  { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-151:null) Seq
14-647496884: Executing request
2013-08-10 00:02:25,457 WARN  [agent.manager.DirectAgentAttache] (DirectAgent-151:null) Seq
14-647496884: Exception caught
java.lang.NullPointerException
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null)
SeqA 10-13769: Processing Seq 10-13769:  { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null)
SeqA 10-13769: Sending Seq 10-13769:  { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null)
SeqA 16-11861: Processing Seq 16-11861:  { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null)
SeqA 16-11861: Sending Seq 16-11861:  { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null)
SeqA 10-13770: Processing Seq 10-13770:  { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null)
SeqA 10-13770: Sending Seq 10-13770:  { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null)
SeqA 16-11862: Processing Seq 16-11862:  { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null)
SeqA 16-11862: Sending Seq 16-11862:  { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }






1. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message