cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailaja Mada (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-4228) [VMWARE]NPE & operation timed out issues while migrating volumes across different Zone wide primary storages
Date Sat, 10 Aug 2013 08:49:49 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sailaja Mada updated CLOUDSTACK-4228:
-------------------------------------

    Fix Version/s: 4.2.0
    
> [VMWARE]NPE & operation timed out issues while migrating volumes across different
Zone wide primary storages
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4228
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Storage Controller, VMware
>    Affects Versions: 4.2.0
>            Reporter: Sailaja Mada
>            Priority: Critical
>             Fix For: 4.2.0
>
>         Attachments: apilog.log, db.dmp, management-server.log
>
>
> Steps:
> 1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster 
> 2. Configured new zone with VMWARE cluster of 2 hosts  with zone wide primary storage

> 3. Added one more Zone wide primary storage to this new zone
> 4. Deploy instance on this zone 
> 5. Add new DATA volume and attach to this instance 
> 6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary storage
1 to second Zone wide primary storage which is added to the same zone 
> Observation:
> 1. NPE & operation timed out issues are observed while migrating volumes across different
Zone wide primary storage's
> 2. It is observed 3 times when tried about 6 times. 
> log & DB dumb is attached . This is a single node management server 
> 2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Waiting some more time because this is the current command
> 2013-08-10 00:02:25,453 INFO  [utils.exception.CSExceptionErrorCode] (StorageManager-Scavenger-4:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
> 2013-08-10 00:02:25,453 WARN  [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Timed out on Seq 18-1413808894:  { Cmd , MgmtId: 90310994128556, via: 18,
Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
> 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: Cancelling.
> 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null)
Seq 18-1413808894: No more commands found
> 2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-4:null)
Unable to send storage pool command to org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2
via 18
> com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host 18 timed
out after 3600
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439)
>         at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987)
>         at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438)
>         at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452)
>         at com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793)
>         at com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039)
>         at com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null)
Seq 14-647496884: Sending  { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
> 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null)
Seq 14-647496884: Executing:  { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
> 2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-151:null)
Seq 14-647496884: Executing request
> 2013-08-10 00:02:25,457 WARN  [agent.manager.DirectAgentAttache] (DirectAgent-151:null)
Seq 14-647496884: Exception caught
> java.lang.NullPointerException
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null)
SeqA 10-13769: Processing Seq 10-13769:  { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null)
SeqA 10-13769: Sending Seq 10-13769:  { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null)
SeqA 16-11861: Processing Seq 16-11861:  { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null)
SeqA 16-11861: Sending Seq 16-11861:  { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null)
SeqA 10-13770: Processing Seq 10-13770:  { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null)
SeqA 10-13770: Sending Seq 10-13770:  { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null)
SeqA 16-11862: Processing Seq 16-11862:  { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null)
SeqA 16-11862: Sending Seq 16-11862:  { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 1. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message