Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5190A18830 for ; Thu, 1 Oct 2015 21:02:29 +0000 (UTC) Received: (qmail 84071 invoked by uid 500); 1 Oct 2015 21:02:29 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 84046 invoked by uid 500); 1 Oct 2015 21:02:29 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 83949 invoked by uid 500); 1 Oct 2015 21:02:29 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 83928 invoked by uid 99); 1 Oct 2015 21:02:29 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2015 21:02:29 +0000 Date: Thu, 1 Oct 2015 21:02:28 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-8808) Successfully registered VHD template is downloaded again due to missing virtualsize property in template.properties MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940408#comment-14940408 ] ASF GitHub Bot commented on CLOUDSTACK-8808: -------------------------------------------- Github user remibergsma commented on the pull request: https://github.com/apache/cloudstack/pull/901#issuecomment-144847280 @karuturi Thanks a lot, works great! LGTM I registered a template with invalid headers, as per CLOUDSTACK-8808. At first everything is normal: screen shot 2015-09-30 at 21 09 29 It then starts installing (and unzipping): screen shot 2015-09-30 at 21 11 22 And eventually disappears from the UI. The logs show `Format is invalid` as expected. Full logs: ``` 2015-10-01 20:55:00,624 WARN [storage.template.TemplateLocation] (pool-1-thread-2:null) Format is invalid 2015-10-01 20:55:00,625 DEBUG [storage.template.TemplateLocation] (pool-1-thread-2:null) Format: VHD size: 21475270656 virtualsize: 0 filename: 4db55a50-1775-3f5d-b982-962eb2d0f282.vhd 2015-10-01 20:55:00,625 DEBUG [storage.template.TemplateLocation] (pool-1-thread-2:null) format, filename cannot be null and size, virtual size should be > 0 2015-10-01 20:55:02,241 DEBUG [storage.template.TemplateLocation] (pool-1-thread-2:null) Remove /mnt/SecStorage/6cc407de-bf89-3f72-8985-1b9df6a63827/template/tmpl/2/201/4db55a50-1775-3f5d-b982-962eb2d0f282.vhd 2015-10-01 20:55:02,302 DEBUG [storage.template.TemplateLocation] (pool-1-thread-2:null) Remove /mnt/SecStorage/6cc407de-bf89-3f72-8985-1b9df6a63827/template/tmpl/2/201/template.properties 2015-10-01 20:55:02,867 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2015-10-01 20:55:02,870 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 3-5925611209712730174: { Ans: , MgmtId: 90520732674659, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"d0965877-d8b8-4e12-a132-703fb7ee6b54","downloadPct":100,"errorString":"Failed post download script: Unable to install due to invalid file format","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/6cc407de-bf89-3f72-8985-1b9df6a63827/template/tmpl/2/201/dnld3817046421300354449tmp_","installPath":"template/tmpl/2/201/4db55a50-1775-3f5d-b982-962eb2d0f282.vhd","templateSize":0,"templatePhySicalSize":0,"checkSum":"49c737f858448e11eb181f250e4efaeb","result":true,"details":"Failed post download script: Unable to install due to invalid file format","wait":0}}] } ``` When someone else reviews this we can merge it and solve another blocker. > Successfully registered VHD template is downloaded again due to missing virtualsize property in template.properties > ------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-8808 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8808 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Secondary Storage > Affects Versions: 4.4.4, 4.6.0 > Environment: Seen on NFS as sec storage > Reporter: Remi Bergsma > Assignee: Rajani Karuturi > Priority: Blocker > > We noticed all of our templates are downloaded again as soon as we restart SSVM, its Cloud service or the management server it connects to. > A scan done by the SSVM (listvmtmplt.sh) returns the template, but it is rejected later (Post download installation was not completed) because (Format is invalid) due to missing virtualSize property in template.properties. > The initial registration did succeed however. I'd either want the registration to fail, or it to succeed. Not first succeed (and spin VMs without a problem) then fail unexpectedly later. > This is the script processing the download: > services/secondary-storage/server/src/org/apache/cloudstack/storage/template/DownloadManagerImpl.java > 759 private List listTemplates(String rootdir) { > 760 List result = new ArrayList(); > 761 > 762 Script script = new Script(listTmpltScr, s_logger); > 763 script.add("-r", rootdir); > For example this becomes: > ==> /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0 > In this log file, it processes the output: > less /var/log/cloud/cloud.out > 2015-09-04 08:39:54,622 WARN [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:null) Post download installation was not completed for /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/2/1607 > This error message is generated here: > services/secondary-storage/server/src/org/apache/cloudstack/storage/template/DownloadManagerImpl.java > > 780 List publicTmplts = listTemplates(templateDir); > 781 for (String tmplt : publicTmplts) { > 782 String path = tmplt.substring(0, tmplt.lastIndexOf(File.separator)); > 783 TemplateLocation loc = new TemplateLocation(_storage, path); > 784 try { > 785 if (!loc.load()) { > 786 s_logger.warn("Post download installation was not completed for " + path); > 787 // loc.purge(); > 788 _storage.cleanup(path, templateDir); > 789 continue; > 790 } > 791 } catch (IOException e) { > 792 s_logger.warn("Unable to load template location " + path, e); > 793 continue; > 794 } > In the logs this message is also seen: > MCCP-ADMIN-1|s-32436-VM CLOUDSTACK: 10:09:17,333 WARN TemplateLocation:196 - Format is invalid > It is generated here: > .//core/src/com/cloud/storage/template/TemplateLocation.java > 192 public boolean addFormat(FormatInfo newInfo) { > 193 deleteFormat(newInfo.format); > 194 > 195 if (!checkFormatValidity(newInfo)) { > 196 s_logger.warn("Format is invalid "); > 197 return false; > 198 } > 199 > 200 _props.setProperty("virtualsize", Long.toString(newInfo.virtualSize)); > 201 _formats.add(newInfo); > 202 return true; > 203 } > This returns false if checkFormatValidity is false. > checkFormatValidity: > 209 protected boolean checkFormatValidity(FormatInfo info) { > 210 return (info.format != null && info.size > 0 && info.virtualSize > 0 && info.filename != null); > 211 } > This returns false if virtualSize is missing in template.properties. And indeed it is missing. > Examples: > Working KVM > root@s-44134-VM:/mnt/SecStorage/884db36b-cd00-3cf0-b812-831e9e50a4b3/template/tmpl/2/2932# cat template.properties > # > #Thu Sep 03 11:41:44 UTC 2015 > filename=125672aa-62c0-30f1-ae1b-153486aa10eb.qcow2 > id=2932 > qcow2.size=1808203776 > public=true > uniquename=2932-2-5c53bbe1-0c32-32a9-a82c-a5eefcb472e5 > qcow2.virtualsize=21474836480 > virtualsize=21474836480 > checksum=4c28fc94a7a9cbe858918a97f3b78294 > hvm=true > description=Centos7-x86_64-Sbp_cis-KVM release 2015-35 build > qcow2=true > qcow2.filename=125672aa-62c0-30f1-ae1b-153486aa10eb.qcow2 > Working XenServer > cat /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/9/603/template.properties > filename=16447323-ae24-4956-ac67-316eed8a2626.vhd > vhd=true > id=603 > vhd.filename=16447323-ae24-4956-ac67-316eed8a2626.vhd > public=false > uniquename=16447323-ae24-4956-ac67-316eed8a2626 > vhd.virtualsize=107374182400 > virtualsize=107374182400 > checksum= > hvm=true > description=570a3650f0-c982-334a-81b1-e8cf961363e4 > vhd.size=69028311552 > size=69028311552 > Not working XenServer > root@s-32435-VM:/mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/2/1607# cat template.properties > # > #Fri Sep 04 09:05:24 UTC 2015 > filename=39794796-3f92-3168-9c4e-a64da5ee06e8.vhd > uniquename=1607-2-0a13984d-6724-3b9f-8bd1-2798ef859167 > size=21475270656 > checksum=6eb85d82cb18032eba0cc8fe8a84a583 > description=Centos7-x86_64-Sbp_cis-XenServer release 2015-35 build > hvm=true > public=true > id=1607 > Confirmed the virtualsize is missing. But why is this successfully registered in the first place? > Steps to reproduce: > - register template > - have it install successfully > - check templates.properties file > sometimes virtualsize is missing > - if it's there, remove it > - restart cloud service on SSVM > - the download will start again > in my case virtualsize is still missing > - this becomes an endless loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)