Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1A02B102A5 for ; Sat, 10 Aug 2013 08:44:03 +0000 (UTC) Received: (qmail 38243 invoked by uid 500); 10 Aug 2013 08:44:03 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 35406 invoked by uid 500); 10 Aug 2013 08:43:57 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 35368 invoked by uid 500); 10 Aug 2013 08:43:55 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 35353 invoked by uid 99); 10 Aug 2013 08:43:54 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Aug 2013 08:43:54 +0000 Date: Sat, 10 Aug 2013 08:43:54 +0000 (UTC) From: "Sailaja Mada (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (CLOUDSTACK-4228) [VMWARE]NPE & operation timed out issues while migrating volumes across different Zone wide primary storages MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Sailaja Mada created CLOUDSTACK-4228: ---------------------------------------- Summary: [VMWARE]NPE & operation timed out issues while migrating volumes across different Zone wide primary storages Key: CLOUDSTACK-4228 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Storage Controller, VMware Affects Versions: 4.2.0 Reporter: Sailaja Mada Priority: Critical Steps: 1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster 2. Configured new zone with VMWARE cluster of 2 hosts with zone wide primary storage 3. Added one more Zone wide primary storage to this new zone 4. Deploy instance on this zone 5. Add new DATA volume and attach to this instance 6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary storage 1 to second Zone wide primary storage which is added to the same zone Observation: 1. NPE & operation timed out issues are observed while migrating volumes across different Zone wide primary storage's 2. It is observed 3 times when tried about 6 times. log & DB dumb is attached . This is a single node management server 2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null) Seq 18-1413808894: Waiting some more time because this is the current command 2013-08-10 00:02:25,453 INFO [utils.exception.CSExceptionErrorCode] (StorageManager-Scavenger-4:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2013-08-10 00:02:25,453 WARN [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null) Seq 18-1413808894: Timed out on Seq 18-1413808894: { Cmd , MgmtId: 90310994128556, via: 18, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] } 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null) Seq 18-1413808894: Cancelling. 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] (StorageManager-Scavenger-4:null) Seq 18-1413808894: No more commands found 2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-4:null) Unable to send storage pool command to org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2 via 18 com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host 18 timed out after 3600 at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430) at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486) at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439) at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987) at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438) at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452) at com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793) at com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039) at com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null) Seq 14-647496884: Sending { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] } 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] (StorageManager-Scavenger-4:null) Seq 14-647496884: Executing: { Cmd , MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] } 2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-151:null) Seq 14-647496884: Executing request 2013-08-10 00:02:25,457 WARN [agent.manager.DirectAgentAttache] (DirectAgent-151:null) Seq 14-647496884: Exception caught java.lang.NullPointerException at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) SeqA 10-13769: Processing Seq 10-13769: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) SeqA 10-13769: Sending Seq 10-13769: { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 16-11861: Processing Seq 16-11861: { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 16-11861: Sending Seq 16-11861: { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null) SeqA 10-13770: Processing Seq 10-13770: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null) SeqA 10-13770: Sending Seq 10-13770: { Ans: , MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) SeqA 16-11862: Processing Seq 16-11862: { Cmd , MgmtId: -1, via: 16, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-14:null) SeqA 16-11862: Sending Seq 16-11862: { Ans: , MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira