Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B4D3C105C7 for ; Mon, 12 Aug 2013 17:04:52 +0000 (UTC) Received: (qmail 14720 invoked by uid 500); 12 Aug 2013 17:04:51 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 14460 invoked by uid 500); 12 Aug 2013 17:04:50 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 14124 invoked by uid 500); 12 Aug 2013 17:04:50 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 14023 invoked by uid 99); 12 Aug 2013 17:04:49 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Aug 2013 17:04:49 +0000 Date: Mon, 12 Aug 2013 17:04:49 +0000 (UTC) From: "Min Chen (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (CLOUDSTACK-4250) [Object_store_refactor] Register ISO fails in multizone environment if downloading to one of the zone fails MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Chen reassigned CLOUDSTACK-4250: ------------------------------------ Assignee: Min Chen > [Object_store_refactor] Register ISO fails in multizone environment if downloading to one of the zone fails > ----------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-4250 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4250 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: ISO, Storage Controller, Template > Affects Versions: 4.2.0 > Environment: Latest build from ACS 4.2 branch. > Storage: NFS for both primary and secondary storage > Reporter: Sanjeev N > Assignee: Min Chen > Priority: Critical > Fix For: 4.2.0 > > Attachments: cloud.dmp, management-server.rar > > > Registering ISO/template with AllZones yes fails If the setup has more than one zone and one of the zones is disabled and SSVM is not running. > Steps to Reproduce: > ================ > 1.Bring up CS with more than one zone say zone1 and zone2 both using NFS for primary and secondary storage > 2.Disable zone2 and stop SSVM in it > 3.Register ISO/Template in zone1 and choose zone= All Zones in UI while registering iso > Result: > ====== > Template download was succeeded in zone1 but it was timed out in zone2. Timeout in downloading template in zone2 deleted the template from both the zones > Observations: > =========== > Log snippet from management server log file: > 2013-08-12 20:58:37,978 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===START=== 10.146.0.20 -- GET command=registerIso&response=json&sessionkey=G1sYAr9SNj4GF7irv5eSdcqDB10%3D&name=all&displayText=all&url=http%3A%2F%2F10.147.28.7%2Ftemplates%2Fvmware%2Fdummy.iso&zoneid=-1&isextractable=true&bootable=true&osTypeId=12&ispublic=true&isfeatured=true&_=1376301850336 > 2013-08-12 20:58:38,171 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is not in store:6, type:Image > 2013-08-12 20:58:38,250 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is already in store:6, type:Image > 2013-08-12 20:58:38,286 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Downloading template to data store 6 > 2013-08-12 20:58:38,313 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Performing image store createTemplate async callback > 2013-08-12 20:58:38,340 DEBUG [cloudstack.storage.RemoteHostEndPoint] (catalina-exec-22:null) Sending command org.apache.cloudstack.storage.command.DownloadCommand to host: 7 > 2013-08-12 20:58:38,344 DEBUG [agent.transport.Request] (catalina-exec-22:null) Seq 7-1199968632: Sending { Cmd , MgmtId: 6851580133501, via: 7, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":true,"description":"all","maxDownloadSizeInBytes":53687091200,"id":220,"resourceType":"TEMPLATE","installPath":"template/tmpl/9/220","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/manasa/secondaryVMw","_role":"Image"}},"url":"http://10.147.28.7/templates/vmware/dummy.iso","format":"ISO","accountId":9,"name":"220-9-41f76fdf-af7e-3d03-91aa-f7fd95fcdf02","secUrl":"nfs://10.147.28.7/export/home/manasa/secondaryVMw","wait":0}}] } > 2013-08-12 20:58:38,350 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is not in store:9, type:Image > 2013-08-12 20:58:38,390 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is already in store:9, type:Image > 2013-08-12 20:58:38,431 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Downloading template to data store 9 > 2013-08-12 20:58:38,447 INFO [storage.endpoint.DefaultEndPointSelector] (catalina-exec-22:null) No running ssvm is found, so command will be sent to LocalHostEndPoint > 2013-08-12 20:58:38,450 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Performing image store createTemplate async callback > 2013-08-12 20:58:38,471 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is not in store:2, type:Image > 2013-08-12 20:58:38,499 DEBUG [storage.image.TemplateDataFactoryImpl] (catalina-exec-22:null) template 220 is already in store:2, type:Image > 2013-08-12 20:58:38,524 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Downloading template to data store 2 > In the above log, store 9 was present in a zone which was in disabled state and SSVM was in stopped state. But still CS initiated download via LocalHostEndPoint > Q: What is LocalHostEndPoint? Is it management server? If yes why do we initiate download via management server? > 2013-08-12 20:58:38,540 DEBUG [storage.image.BaseImageStoreDriverImpl] (catalina-exec-22:null) Performing image store createTemplate async callback > 2013-08-12 20:58:38,556 DEBUG [cloudstack.storage.RemoteHostEndPoint] (catalina-exec-22:null) Sending command org.apache.cloudstack.storage.command.DownloadCommand to host: 14 > 2013-08-12 20:58:38,559 DEBUG [agent.transport.Request] (catalina-exec-22:null) Seq 14-670765384: Sending { Cmd , MgmtId: 6851580133501, via: 14, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":true,"description":"all","maxDownloadSizeInBytes":53687091200,"id":220,"resourceType":"TEMPLATE","installPath":"template/tmpl/9/220","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/manasa/secondaryxen","_role":"Image"}},"url":"http://10.147.28.7/templates/vmware/dummy.iso","format":"ISO","accountId":9,"name":"220-9-41f76fdf-af7e-3d03-91aa-f7fd95fcdf02","secUrl":"nfs://10.147.28.7/export/home/manasa/secondaryxen","wait":0}}] } > 2013-08-12 20:58:38,594 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END=== 10.146.0.20 -- GET command=registerIso&response=json&sessionkey=G1sYAr9SNj4GF7irv5eSdcqDB10%3D&name=all&displayText=all&url=http%3A%2F%2F10.147.28.7%2Ftemplates%2Fvmware%2Fdummy.iso&zoneid=-1&isextractable=true&bootable=true&osTypeId=12&ispublic=true&isfeatured=true&_=1376301850336 > 2013-08-12 20:58:42,059 DEBUG [agent.transport.Request] (AgentManager-Handler-12:null) Seq 7-1199968632: Processing: { Ans: , MgmtId: 6851580133501, via: 7, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"edf48a3a-24a2-4ea5-8863-87cba1a31971","downloadPct":100,"errorString":"Download success, starting install ","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/0e8da06e-0788-3efb-86a6-b0705a2205d3/template/tmpl/9/220/dnld6918760524895984102tmp_","installPath":"template/tmpl/9/220/220-9-41f76fdf-af7e-3d03-91aa-f7fd95fcdf02.iso","templateSize":0,"templatePhySicalSize":0,"checkSum":"18425b903c6969ed63f328ef4fd346af","result":true,"details":"Download success, starting install ","wait":0}}] } > 2013-08-12 20:58:42,061 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 14-670765384: Processing: { Ans: , MgmtId: 6851580133501, via: 14, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"4dafa883-a1bf-4693-a463-049120455173","downloadPct":100,"errorString":"Download success, starting install ","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1/template/tmpl/9/220/dnld2814394365859359939tmp_","installPath":"template/tmpl/9/220/220-9-41f76fdf-af7e-3d03-91aa-f7fd95fcdf02.iso","templateSize":0,"templatePhySicalSize":0,"checkSum":"18425b903c6969ed63f328ef4fd346af","result":true,"details":"Download success, starting install ","wait":0}}] } > 2013-08-12 20:58:42,623 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null) SeqA 4-36208: Processing Seq 4-36208: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } > 2013-08-12 20:58:42,632 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null) SeqA 4-36208: Sending Seq 4-36208: { Ans: , MgmtId: 6851580133501, via: 4, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2013-08-12 20:58:43,257 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) StorageCollector is running... > 2013-08-12 20:58:43,817 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 14-670765385: Received: { Ans: , MgmtId: 6851580133501, via: 14, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > 2013-08-12 20:58:43,934 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-1199968633: Received: { Ans: , MgmtId: 6851580133501, via: 7, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > 2013-08-12 20:58:43,938 INFO [storage.endpoint.DefaultEndPointSelector] (StatsCollector-1:null) No running ssvm is found, so command will be sent to LocalHostEndPoint > Before deleting ISO from CS I could see the following timeout messages: > 2013-08-12 20:59:38,450 DEBUG [storage.download.DownloadListener] (Timer-25:null) timeout: transitioning to download error state, currstate=NOT_DOWNLOADED, TEMPLATE: 220 at host 0 > 2013-08-12 20:59:38,450 DEBUG [storage.image.BaseImageStoreDriverImpl] (Timer-25:null) Performing image store createTemplate async callback > 2013-08-12 20:59:38,543 WARN [storage.download.DownloadListener] (Timer-25:null) Entering download error state: timeout waiting for response from storage host, TEMPLATE: 220 at host 0 > Attaching management server log file and cloud DB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira