cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Repton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-8808) Successfully registered VHD template is downloaded again due to missing virtualsize property in template.properties
Date Mon, 28 Sep 2015 14:34:04 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933368#comment-14933368
] 

Andrew Repton commented on CLOUDSTACK-8808:
-------------------------------------------

Hi Rajani,

I believe you are correct. It's slightly more complex for us as the code has changed from
GA-4.4.4 and the code for us that failed was:

   try {
            strm = new FileInputStream(file);
            strm.skip(file.length() - vhdFooterSize + vhdFooterCreatorAppOffset);
            strm.read(creatorApp);
            strm.skip(vhdFooterCurrentSizeOffset - vhdFooterCreatorVerOffset);
            strm.read(currentSize);
        } catch (Exception e) {
            s_logger.warn("Unable to read vhd file " + file.getAbsolutePath(), e);
            throw new CloudRuntimeException("Unable to read vhd file " + file.getAbsolutePath()
+ ": " + e);
        } finally {
            if (strm != null) {
                try {
                    strm.close();
                } catch (IOException e) {
                }
            }
        }

Rather than the current master:
     try {
            info.virtualSize = getVirtualSize(vhdFile);
        } catch (IOException e) {
            s_logger.error("Unable to get the virtual size for " + vhdPath);
            return null;
        }

So we had no error outputted at all as it proceeded onto the Finally statement. I'm not certain
if proceeding with the actual size is a good idea in this circumstance though, as it means
that Cloudstack cannot correctly identify the sparsed template size, which will lead to incorrect
disk usage statistics.

The reproducible steps for this are:

1. Install qemu-img at any version less than 2.0.0 (current version in centos-updates repository
for CentOS 7)
2. Generate a qcow2 image
3. Run the following command: 
qemu-img convert -f qcow2 -O vpc template.qcow2 template.vhd
4. Upload to Cloudstack 4.4.4. The upload will succeed but the Size will be blank in the GUI
and it will be missing the virtualsize from the template.properties file on the Secondary
Storage VM. Starting the VM should work fine (provided the Xen drivers were added to the initramfs
as part of building the template)

The size reported using ls on a Linux system will show it as the 'full size', rather than
the sparsed size. However, to tools that can recognise sparsed files it will show up as much
smaller. To Cloudstack however, you have just used up the full size of the underlying storage.
Debatable how important this is.

Any version of qemu-img that is version 2.0.0 or higher correctly converts the qcow2 to a
VHD file and this bug does not occur.

(I should mention I'm a colleague of Remi's, and unfortunately the templates I have all have
confidential information on them so I can't give you one. If it's really required to proceed
please let me know and I can follow the above steps to try and build you one)

Regards,

Andy

> Successfully registered VHD template is downloaded again due to missing virtualsize property
in template.properties
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-8808
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8808
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Secondary Storage
>    Affects Versions: 4.4.4, 4.6.0
>         Environment: Seen on NFS as sec storage
>            Reporter: Remi Bergsma
>            Assignee: Rajani Karuturi
>            Priority: Blocker
>
> We noticed all of our templates are downloaded again as soon as we restart SSVM, its
Cloud service or the management server it connects to.
> A scan done by the SSVM (listvmtmplt.sh) returns the template, but it is rejected later
(Post download installation was not completed) because (Format is invalid) due to missing
virtualSize property in template.properties.
> The initial registration did succeed however. I'd either want the registration to fail,
or it to succeed. Not first succeed (and spin VMs without a problem) then fail unexpectedly
later.
> This is the script processing the download:
> services/secondary-storage/server/src/org/apache/cloudstack/storage/template/DownloadManagerImpl.java
>  759     private List<String> listTemplates(String rootdir) {                 
       
>  760         List<String> result = new ArrayList<String>();             
             
>  761                                                                                
 
>  762         Script script = new Script(listTmpltScr, s_logger);                    
 
>  763         script.add("-r", rootdir);   
> For example this becomes:
> ==> /usr/local/cloud/systemvm/scripts/storage/secondary/listvmtmplt.sh -r /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0
> In this log file, it processes the output:
> less /var/log/cloud/cloud.out
> 2015-09-04 08:39:54,622 WARN  [storage.template.DownloadManagerImpl] (agentRequest-Handler-1:null)
Post download installation was not completed for /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/2/1607
> This error message is generated here:
> services/secondary-storage/server/src/org/apache/cloudstack/storage/template/DownloadManagerImpl.java
>  
> 780         List<String> publicTmplts = listTemplates(templateDir);           
      
>  781         for (String tmplt : publicTmplts) {                                    
 
>  782             String path = tmplt.substring(0, tmplt.lastIndexOf(File.separator));

>  783             TemplateLocation loc = new TemplateLocation(_storage, path);       
 
>  784             try {                                                              
 
>  785                 if (!loc.load()) {                                             
 
>  786                     s_logger.warn("Post download installation was not completed
for " + path);
>  787                     // loc.purge();                                            
 
>  788                     _storage.cleanup(path, templateDir);                       
 
>  789                     continue;                                                  
 
>  790                 }                                                              
 
>  791             } catch (IOException e) {                                          
 
>  792                 s_logger.warn("Unable to load template location " + path, e);  
 
>  793                 continue;                                                      
 
>  794             } 
> In the logs this message is also seen:
> MCCP-ADMIN-1|s-32436-VM CLOUDSTACK: 10:09:17,333  WARN TemplateLocation:196 - Format
is invalid 
> It is generated here:
> .//core/src/com/cloud/storage/template/TemplateLocation.java
> 192    public boolean addFormat(FormatInfo newInfo) {                               
> 193         deleteFormat(newInfo.format);                                           

> 194                                                                                 

> 195         if (!checkFormatValidity(newInfo)) {                                    

> 196             s_logger.warn("Format is invalid ");                                

> 197             return false;                                                       

> 198         }                                                                       

> 199                                                                                 

> 200         _props.setProperty("virtualsize", Long.toString(newInfo.virtualSize));  

> 201         _formats.add(newInfo);                                                  

> 202         return true;                                                            

> 203     }                          
> This returns false if checkFormatValidity is false.
> checkFormatValidity:
> 209     protected boolean checkFormatValidity(FormatInfo info) {                    

> 210         return (info.format != null && info.size > 0 && info.virtualSize
> 0 && info.filename != null);
> 211     } 
> This returns false if virtualSize is missing in template.properties. And indeed it is
missing.
> Examples:
> Working KVM
> root@s-44134-VM:/mnt/SecStorage/884db36b-cd00-3cf0-b812-831e9e50a4b3/template/tmpl/2/2932#
cat template.properties
> #
> #Thu Sep 03 11:41:44 UTC 2015
> filename=125672aa-62c0-30f1-ae1b-153486aa10eb.qcow2
> id=2932
> qcow2.size=1808203776
> public=true
> uniquename=2932-2-5c53bbe1-0c32-32a9-a82c-a5eefcb472e5
> qcow2.virtualsize=21474836480
> virtualsize=21474836480
> checksum=4c28fc94a7a9cbe858918a97f3b78294
> hvm=true
> description=Centos7-x86_64-Sbp_cis-KVM release 2015-35 build
> qcow2=true
> qcow2.filename=125672aa-62c0-30f1-ae1b-153486aa10eb.qcow2
> Working XenServer
> cat /mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/9/603/template.properties
> filename=16447323-ae24-4956-ac67-316eed8a2626.vhd
> vhd=true
> id=603
> vhd.filename=16447323-ae24-4956-ac67-316eed8a2626.vhd
> public=false
> uniquename=16447323-ae24-4956-ac67-316eed8a2626
> vhd.virtualsize=107374182400
> virtualsize=107374182400
> checksum=
> hvm=true
> description=570a3650f0-c982-334a-81b1-e8cf961363e4
> vhd.size=69028311552
> size=69028311552
> Not working XenServer
> root@s-32435-VM:/mnt/SecStorage/ee8633dd-5dbd-39a3-b3ea-801ca0a20da0/template/tmpl/2/1607#
cat template.properties 
> #
> #Fri Sep 04 09:05:24 UTC 2015
> filename=39794796-3f92-3168-9c4e-a64da5ee06e8.vhd
> uniquename=1607-2-0a13984d-6724-3b9f-8bd1-2798ef859167
> size=21475270656
> checksum=6eb85d82cb18032eba0cc8fe8a84a583
> description=Centos7-x86_64-Sbp_cis-XenServer release 2015-35 build
> hvm=true
> public=true
> id=1607
> Confirmed the virtualsize is missing. But why is this successfully registered in the
first place?
> Steps to reproduce:
> - register template
> - have it install successfully
> - check templates.properties file
>   sometimes virtualsize is missing
> - if it's there, remove it
> - restart cloud service on SSVM
> - the download will start again
>   in my case virtualsize is still missing
> - this becomes an endless loop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message