cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: Strange size of template from snapshot on XenServer
Date Fri, 10 Nov 2017 11:55:02 GMT
Anshul,

I will not touch the merit of documentation in code, for now, this is
another discussion.  And regarding the “optimization”, you brought it up,
whenever I saw you discussing this issue, you would talk about some sort of
“optimization”… see
https://github.com/apache/cloudstack/pull/1124#issuecomment-326550060. At
least for me, that was quite confusing, and I was not seeing what you were
talking about.

If you diff the files, what broke your “solution” was the removal of the
following lines:

> EndPoint ep = endPointSelector.selectHypervisorHost(new ZoneScope(host.
> getDataCenterId()));
> host = hostDao.findById(ep.getId());
>

The conditional checks were maintained. We only removed something that we
seemed logicless.

With PR https://github.com/apache/cloudstack/pull/2315, I did not add
anything extra. I only re-worked the method to make it clear and more
readable (removed the if-inception). The only point I changed is that,
instead of using “selectOneHypervisorHostByZone”, I create a specific
method for that, which selects a host according to the volume of the
snapshot.

On Fri, Nov 10, 2017 at 8:21 AM, Anshul Gangwar <
anshul.gangwar@accelerite.com> wrote:

> Rafael, you are still misunderstanding the fix “
> 2c4ea503f92bcf9c611f409d5cdecb”. That has only fixed the issue due to
> check for specific version i.e. before “2c4ea503f92bcf9c611f409d5cdecb”
> things were fine for xenserver 6.2 but not working for 6.5. Commit doesn’t
> mention anything about optimization. Other than version change it reverts
> the previous commit which is introducing hypervisor specific check in core
> code.
>
> Regarding documentation, “git history”is the best documentation. Code
> documentation may not make sense after some changes as it is often ignored.
>
> See the PR for more comments.
>
> Regards,
> Anshul
>
> On 10/11/17, 3:14 AM, "Rafael Weingärtner" <rafaelweingartner@gmail.com>
> wrote:
>
>     Anshul and Lotic, after debugging and inspecting code for quite a
> while I
>     understood the complete picture and created a solution for it. The PR
> with
>     the solution is found athttps://github.com/apache/cloudstack/pull/2315
> .
>     This was a clear example of what I constantly tell people here, code
> needs
>     to be clear, concise, well tested and documented. Here follows some
>     explanation regarding the removal of some code introduced with commit
>     “2c4ea503f92bcf9c611f409d5cdecb”.
>
>     With commit “8caf52c” Daan and I removed a small piece of code
> introduced
>     by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to make
> much
>     sense. We did not remove everything; we only removed the part that was
>     looking for a random host in the zone to execute the command. As we
>     (Anshul) exchanged some messages in some other PR, the code introduced
> in
>     “2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that do
> not have
>     other hypervisor types if they are deployed with XenServer clusters.It
> was
>     being created a limitation in ACS that should not exist. I stress again
>     that this only happened for the lack of documentation and clear coding.
>     For instance, when I read the commit “2c4ea503f92bcf9c611f409d5cdecb”,
> it
>     says it introduced an “optimization”, then I assumed that the process
>     executed with our code base before commit “
> 2c4ea503f92bcf9c611f409d5cdecb”
>     is working, but not as fast or with as much quality as the code with
>     “2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the case; the
> code
>     “2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it is, in
>     fact, fixing/creating a workflow to create templates from snapshots in
>     XenServer deployments.
>
>     The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only
> tackling
>     the problem for CloudStack deployments that use single hypervisor types
>     (restricted to XenServers host in the same zone; this means, it was not
>     expecting to have multiple hypervisors types in the same zone).
>     Additionally, the lack of information regarding that solution
>     (documentation, test cases and description in PRs and Jira ticket) led
> the
>     code to be removed in #1124 after a long discussion and analysis in
> #1056.
>     That piece of code seemed logicless.  It would receive a hostId and
> then
>     change that hostId for other hostId of the zone without doing any
> check; it
>     was not even checking the hypervisor and storage in which the host was
>     connected to.
>
>     The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots
>     that are taken in XenServer. This means, we do not take a complete
>     snapshot, but a partial one that contains only the modified data. This
>     requires rebuilding the VHD hierarchy when creating a template out of
> the
>     snapshot. The point is that the first hostId received is not a hostId,
> but
>     a system VM ID(SSVM). That is why the code in #1176 fixed the problem
> for
>     some deployment scenarios, but would cause problems for scenarios
> where we
>     have multiple hypervisors in the same zone. We need to execute the
> creation
>     of the VHD that represents the template in the hypervisor, so the VHD
> chain
>     can be built using the parent links.
>
>     The PR #2315 changes the behavior of
>     com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long,
> Command).
>     From now on we replace the hostId that is intended to execute the “copy
>     command” that will create the VHD of the template according to some
>     conditions that were already in place. The idea is that starting with
>     XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the
>     hypervisor host and not from the SSVM. Moreover, the method was
> improved
>     making it more readable; it was also created test cases assuring that
> from
>     XenServer 6.2.0 hotFix ESP1004 and upward versions we change the hostId
>     that will be used to execute the “copy command”.
>
>     Furthermore, we are not selecting a random host from a zone anymore. A
> new
>     method was introduced in the HostDao object called
>     “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this
> method
>     we look for a host that is in the cluster that is using the storage
> pool
>     where the volume from which the Snapshot is taken of resides. By doing
>     this, we guarantee that the host that is connected to the primary
> storage
>     where all of the snapshots parent VHDs are stored is used to create the
>     template.
>
>     Also, I opened the PR only against master. If someone needs the fix in
> any
>     other version we can cherry pick to 4.9.2.0.
>
>     On Wed, Nov 8, 2017 at 9:41 AM, Anshul Gangwar <
>     anshul.gangwar@accelerite.com> wrote:
>
>     > Rafael, In ACS 4.5.2 you are facing issue due to XenServer version
> (was
>     > only working on xenserver 6.2 with some patches) which got fixed in
> 4.6
>     > with commit 2c4ea503f92bcf9c611f409d5cdecb42b0115b69. Now with code
> clean
>     > up in commit “8a3fd10” it’s broken. During storage refactoring
> changes in
>     > commit “8caf52c” optimization was done to send CopyCommand to
> Hypervisor
>     > resource instead of SSVM. Now with commit “8a3fd10” it’s again going
> to
>     > SSVM so it’s working fine for first snapshot but broken for linked
> snapshot.
>     >
>     > Regards,
>     > Anshul
>     >
>     > On 04/11/17, 7:20 PM, "Rafael Weingärtner" <
> rafaelweingartner@gmail.com>
>     > wrote:
>     >
>     >     Anshul, I tried with ACS 4.5.2 where commit “8a3fd10” is not
> present.
>     > This
>     >     is the commit where Daan and I removed some code that seemed
> (is? at
>     > least
>     >     for me) logicless (https://github.com/apache/
> cloudstack/pull/1124 and
>     >     https://github.com/apache/cloudstack/pull/1056).
>     >
>     >     I tested with ACS 4.5.2, which I have in a test environment. I
> have ACS
>     >     4.5.2, NFS as the primary storage, and XenServer 6.5 as hosts. I
>     > managed to
>     >     reproduce the problem as Lotic is having.
>     >
>     >     I spin up a VM, then I created a file of 2GB in this VM, and
> later I
>     > took a
>     >     snapshot of the root volume of it (the VM only has the root
> volume).
>     > The
>     >     VHD “687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd” is the first
> snapshot
>     > of the
>     >     root volume. If I create a template from this snapshot,
> everything
>     > works. I
>     >     created a new VM from the template of the first snapshot, and it
>     > worked,
>     >     and the 2GB file was there. Then, I returned for the first VM
> (from
>     > which I
>     >     took the snapshot), I created another 1GB file. Then I stopped
> the VM
>     > again
>     >     and took the second snapshot (27d37d21-13a6-488d-b2f0-
>     > 0ac54236b83a.vhd).
>     >     The interesting thing here is that the file size reduced!? And I
> also
>     >     noticed that there is no hierarchy in the VHDs files. I tried to
> start
>     > a VM
>     >     from a template that I created with the second snapshot, and it
> did not
>     >     work as the problem Lotic is having.
>     >
>     >     [root@xenserver-1 ~]# ls -lah
>     >     > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
> snapshots/2/8/
>     >     > total 6.9G
>     >     > drwxr-xr-x 2 root root 4.0K Nov  4 10:33 .
>     >     > drwxr-xr-x 3 root root 4.0K Nov  4 09:49 ..
>     >     > -rw-r--r-- 1 root root 3.2G Nov  4 10:38
>     >     > 27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
>     >     > -rw-r--r-- 1 root root 3.7G Nov  4 09:55
>     >     > 687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
>     >     > [root@xenserver-1 ~]# vhd-util scan -f -a -p
>     >     > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
>     > snapshots/2/8/687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
>     >     > vhd=687f86f2-1aee-4e90-a64a-3b3c28376fcd.vhd
> capacity=21474836480
>     >     > size=3916767744 hidden=0 parent=none
>     >     > [root@xenserver-1 ~]# vhd-util scan -f -a -p
>     >     > /var/cloud_mount/963e9072-0fa1-3448-bf48-ffde1f522104/
>     > snapshots/2/8/27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
>     >     > vhd=27d37d21-13a6-488d-b2f0-0ac54236b83a.vhd
> capacity=21474836480
>     >     > size=3420873216 hidden=0 parent=none
>     >     >
>     >
>     >
>     >     Was it not working way before “8a3fd10”?! I think we will need
> help
>     > from
>     >     more folks here to see if this is a problem that only affects
> XenServer
>     >     deployments. I will also test on my 4.9.2.0 environment again to
> see
>     > if I
>     >     miss anything.
>     >
>     >     On Thu, Nov 2, 2017 at 6:30 AM, Anshul Gangwar <
>     >     anshul.gangwar@accelerite.com> wrote:
>     >
>     >     > Hi Lotic,
>     >     >
>     >     > Can you try with any release between 4.6 to 4.8 and see if
> this bug
>     > is
>     >     > there?
>     >     >
>     >     > Basically, any release which contains commit
>     > 2c4ea503f92bcf9c611f409d5cdecb42b0115b69
>     >     > and missing commit 8a3fd10.
>     >     >
>     >     > On 02/11/17, 1:14 AM, "Lotic Lists" <lists@lotic.com.br>
> wrote:
>     >     >
>     >     >     Rafael, I know and appreciate your help :)
>     >     >
>     >     >     I recognize all effort of CloudStack community.
>     >     >     I think all admins use snapshot to backup your virtual
> disks, I
>     > think
>     >     > it is not a isolated problem. Imagine a crash on your storage
> and
>     > you need
>     >     > convert your snapshots to templates.
>     >     >
>     >     >     Thanks
>     >     >     Marcelo
>     >     >
>     >     >     -----Original Message-----
>     >     >     From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com]
>     >     >     Sent: quarta-feira, 1 de novembro de 2017 17:17
>     >     >     To: dev@cloudstack.apache.org
>     >     >     Subject: Re: Strange size of template from snapshot on
> XenServer
>     >     >
>     >     >     Well, I could not reproduce the problem in ACS 4.9.2.0. So,
>     > before
>     >     > fixing anything, the person that might take on this should
> first
>     > find the
>     >     > root of the problem.
>     >     >
>     >     >     I understand what you are saying that it is a critical
> problem,
>     > but
>     >     > you have to understand that this is an open-source and free
> software.
>     >     >
>     >     >     Having said that, ALL of my work here has been pro-bono;
> even
>     > though I
>     >     > am a committer and PMC, I am not willing to dive deeper into
> this
>     > issue now
>     >     > as it does not affect me. It does not mean that I would not
> help
>     > people if
>     >     > I have time, as you can notice whenever I have some free time
> I try
>     > to be
>     >     > around and guide or check things for other people in this list.
>     >     >
>     >     >     If this is a huge problem for you right now, I suggest you
>     > looking for
>     >     > companies that provide enterprise support for ACS.
>     >     >
>     >     >     On Wed, Nov 1, 2017 at 7:35 PM, Lotic Lists <
> lists@lotic.com.br>
>     >     > wrote:
>     >     >
>     >     >     > Hi Rafael
>     >     >     >
>     >     >     > Tested on ACS 4.5.2.2, 4.9.2.0, 4.9.3.0 and 4.10. All
> versions
>     > with
>     >     >     > same issue
>     >     >     >
>     >     >     > Description and steps to reproduce
>     >     >     > https://issues.apache.org/jira/browse/CLOUDSTACK-10128
>     >     >     >
>     >     >     > Again, it is a critical problem for production
> environments.
>     >     >     >
>     >     >     > Regards
>     >     >     > Marcelo
>     >     >     >
>     >     >     > -----Original Message-----
>     >     >     > From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com]
>     >     >     > Sent: terça-feira, 31 de outubro de 2017 17:10
>     >     >     > To: dev@cloudstack.apache.org
>     >     >     > Subject: Re: Strange size of template from snapshot on
>     > XenServer
>     >     >     >
>     >     >     > Yes, it is a big problem. I have not checked the code
> that
>     > creates
>     >     > the
>     >     >     > template, but I am not sure if the problem you are
>     > experiencing is
>     >     >     > related to 9025. It might be a problem in some other
> place.
>     >     >     >
>     >     >     > My suggestion is for you to create a Jira ticket
> detailing the
>     >     >     > situation and wait until someone fixes it. Have you
> tested on
>     > ACS
>     >     >     > 4.10? If 4.9.2.0 works, it might be something that went
> only
>     > into
>     >     > 4.9.3.0.
>     >     >     >
>     >     >     > On Tue, Oct 31, 2017 at 8:04 PM, Lotic Lists <
>     > lists@lotic.com.br>
>     >     > wrote:
>     >     >     >
>     >     >     > > I tested on a fresh environment, acs 4.9.3.0,
> xenserver 6.5
>     > with
>     >     > all
>     >     >     > > patches and primary storage NFS, we have same problem
>     >     >     > >
>     >     >     > > - Snapshot files
>     >     >     > > [root@acs01 203]# ls -lh /exports/secondary/snapshots/
> 2/6/
>     > total
>     >     >     > > 2.8G
>     >     >     > > -rw-r--r-- 1 root root 1.7G Oct 31 16:48
>     >     >     > > cad96f0c-de49-4255-b0a0-9aea5e2297cb.vhd
>     >     >     > > <-- snap-base
>     >     >     > > -rw-r--r-- 1 root root 1.1G Oct 31 16:49
>     >     >     > > fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
>     >     >     > > <-- snap-diff
>     >     >     > > https://i.imgur.com/bWupsAS.png
>     >     >     > >
>     >     >     > > [root@acs01 203]# md5sum /exports/secondary/snapshots/
>     >     >     > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
>     >     >     > > 17eec8d4f6d4c128b34b9e2e1876ceb7
>     > /exports/secondary/snapshots/
>     >     >     > > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd
>     >     >     > >
>     >     >     > > - Template files
>     >     >     > > [root@acs01 203]# ls -lh /exports/secondary/template/
>     > tmpl/2/203
>     >     >     > > total 2.0G
>     >     >     > > -rw-r--r-- 1 root root 1.1G Oct 31 16:50
>     > e9def45d-1b9e-4c2e-b843-
>     >     >     > > 852fe40f00b2.vhd
>     >     >     > > -rw-r--r-- 1 root root  303 Oct 31 16:50
> template.properties
>     >     >     > >
>     >     >     > > [root@acs01 203]# md5sum /exports/secondary/template/
>     >     >     > > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd
>     >     >     > > 17eec8d4f6d4c128b34b9e2e1876ceb7
>     > /exports/secondary/template/
>     >     >     > > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd
>     >     >     > >
>     >     >     > > It's a huge problem!!! I think issue 9025 is not
> resolved
>     >     >     > >
>     >     >     > > Regards
>     >     >     > > Marcelo
>     >     >     > >
>     >     >     > > -----Original Message-----
>     >     >     > > From: Rafael Weingärtner [mailto:rafaelweingartner@
> gmail.com
>     > ]
>     >     >     > > Sent: terça-feira, 31 de outubro de 2017 16:00
>     >     >     > > To: users@cloudstack.apache.org
>     >     >     > > Subject: Re: Strange size of template from snapshot on
>     > XenServer
>     >     >     > >
>     >     >     > > Well, reading the description of the issue it seems to
> be
>     > related
>     >     > to
>     >     >     > > the problem you are describing. However, I compared the
>     > classes
>     >     > that
>     >     >     > > were changed with PR
>     >     >     > > (https://github.com/apache/cloudstack/pull/1176)
>     >     >     > > and they are the same in ACS 4.9.2.0 and 4.9.3.0.
>     >     >     > >
>     >     >     > > On Tue, Oct 31, 2017 at 6:36 PM, Lotic Lists <
>     > lists@lotic.com.br>
>     >     > wrote:
>     >     >     > >
>     >     >     > > > Hi Rafael, thanks for test.
>     >     >     > > >
>     >     >     > > > I found the issue
>     >     >     > > > https://issues.apache.org/
> jira/browse/CLOUDSTACK-9025
>     >     >     > > > I think is a same case. I will up a new lab
> environment
>     > now for
>     >     >     > > > simulate the issue
>     >     >     > > >
>     >     >     > > > Regards.
>     >     >     > > > Marcelo
>     >     >     > > >
>     >     >     > > > -----Original Message-----
>     >     >     > > > From: Rafael Weingärtner [mailto:rafaelweingartner@
>     > gmail.com]
>     >     >     > > > Sent: segunda-feira, 30 de outubro de 2017 12:08
>     >     >     > > > To: users@cloudstack.apache.org
>     >     >     > > > Subject: Re: Strange size of template from snapshot
> on
>     > XenServer
>     >     >     > > >
>     >     >     > > > I just did, and the size of the template is the size
> of
>     > the root
>     >     >     > > > disk
>     >     >     > > > + the 1GB file I created in the test vm.
>     >     >     > > >
>     >     >     > > > The system I tested this is an ACS 4.9.2, XenServer
> 6.5,
>     > Primary
>     >     >     > > > Stg ISCSI
>     >     >     > > >
>     >     >     > > > On Mon, Oct 30, 2017 at 10:20 AM, Lotic Lists <
>     >     > lists@lotic.com.br>
>     >     >     > > wrote:
>     >     >     > > >
>     >     >     > > > > Good morning
>     >     >     > > > >
>     >     >     > > > > Guys, who could execute the test below?
>     >     >     > > > >
>     >     >     > > > > 1. Create a manual snapshot of a volume 2. Create
> a file
>     > (1GB)
>     >     >     > > > > with dd on VM 3. Create second snapshot of a
> volume 4.
>     > Convert
>     >     >     > > > > latest snapshot to a template 5. Verify the size
> of vhd
>     > of new
>     >     >     > > > > template in secondary storage.
>     >     >     > > > >
>     >     >     > > > > Here the template have 1GB, exactly the size of
> latest
>     >     > snapshot.
>     >     >     > > > >
>     >     >     > > > > ACS 4.9.3, XenServer 6.5, Primary Stg ISCSI
>     >     >     > > > >
>     >     >     > > > >
>     >     >     > > > > -----Original Message-----
>     >     >     > > > > From: Lotic Lists [mailto:lists@lotic.com.br]
>     >     >     > > > > Sent: quinta-feira, 26 de outubro de 2017 15:03
>     >     >     > > > > To: users@cloudstack.apache.org
>     >     >     > > > > Subject: RE: Strange size of template from
> snapshot on
>     >     > XenServer
>     >     >     > > > >
>     >     >     > > > > I test the same case on other environment, create
>     > template from
>     >     >     > > > > snapshot, acs 4.9.2.0 and XenServer 6.5/iscsi
>     >     >     > > > >
>     >     >     > > > > 1. Was created two snapshots
>     >     >     > > > >
>     >     >     > > > > # ls -lh /var/cloud_mount/e5921d9a-
>     >     > 70b0-3501-9fda-6097e721fffb/
>     >     >     > > > > snapshots/131/617/
>     >     >     > > > > total 3.0G
>     >     >     > > > > -rw-r--r-- 1 4294967294 root  13G Oct 26 13:46
>     >     >     > > > > 42c7c51d-4aa7-40d2-b5eb- cb56fc963974.vhd
>     >     >     > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:42
>     >     >     > > > > 53b33a5a-a7d9-4efa-b126- 38eec5482b05.vhd
>     >     >     > > > >
>     >     >     > > > > https://i.imgur.com/cSHZAJ3.png
>     >     >     > > > >
>     >     >     > > > > 2. create a template from latest snapshot
>     >     >     > > > >
>     >     >     > > > > # ls -lh
>     >     >     > > > > total 37M
>     >     >     > > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:43
>     >     >     > > > > 637604a6-3085-444e-ba09- b9aab11a2c16.vhd
>     >     >     > > > > -rw-r--r-- 1 4294967294 root  303 Oct 26 14:43
>     >     >     > > > > template.properties
>     >     >     > > > >
>     >     >     > > > > # cat template.properties
>     >     >     > > > > #Thu Oct 26 16:43:54 UTC 2017
>     >     >     > > > > filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd
>     >     >     > > > > id=1
>     >     >     > > > > vhd=true
>     >     >     > > > > vhd.filename=637604a6-3085-
> 444e-ba09-b9aab11a2c16.vhd
>     >     >     > > > > public=true
>     >     >     > > > > uniquename=637604a6-3085-444e-ba09-b9aab11a2c16
>     >     >     > > > > vhd.virtualsize=21474836480
>     >     >     > > > > virtualsize=21474836480
>     >     >     > > > > hvm=
>     >     >     > > > > vhd.size=1197752832
>     >     >     > > > > size=1197752832
>     >     >     > > > >
>     >     >     > > > > # md5sum /var/cloud_mount/e5921d9a-
>     >     > 70b0-3501-9fda-6097e721fffb/
>     >     >     > > > > snapshots/131/617/53b33a5a-
> a7d9-4efa-b126-38eec5482b05.
>     > vhd
>     >     >     > > > > 9d2078a6f41ca5e1eab824848c06df53
>     > /var/cloud_mount/e5921d9a-
>     >     >     > > > > 70b0-3501-9fda-6097e721fffb/
> snapshots/131/617/53b33a5a-
>     >     >     > > > > a7d9-4efa-b126-38eec5482b05.vhd
>     >     >     > > > >
>     >     >     > > > > # md5sum /var/cloud_mount/e5921d9a-
>     >     > 70b0-3501-9fda-6097e721fffb/
>     >     >     > > > > template/tmpl/2/294/637604a6-
>     > 3085-444e-ba09-b9aab11a2c16.vhd
>     >     >     > > > > 9d2078a6f41ca5e1eab824848c06df53
>     > /var/cloud_mount/e5921d9a-
>     >     >     > > > > 70b0-3501-9fda-6097e721fffb/
>     > template/tmpl/2/294/637604a6-
>     >     >     > > > > 3085-444e-ba09-b9aab11a2c16.vhd
>     >     >     > > > >
>     >     >     > > > > ACS not merge files of snapshots in a single file
> for
>     > template,
>     >     >     > > > > I think it's a bug.
>     >     >     > > > >
>     >     >     > > > >
>     >     >     > > > > -----Original Message-----
>     >     >     > > > > From: Rafael Weingärtner [mailto:
> rafaelweingartner@
>     > gmail.com]
>     >     >     > > > > Sent: quarta-feira, 25 de outubro de 2017 22:51
>     >     >     > > > > To: users@cloudstack.apache.org
>     >     >     > > > > Subject: Re: Strange size of template from
> snapshot on
>     >     > XenServer
>     >     >     > > > >
>     >     >     > > > > It is easy to test, get one of these files and try
> to
>     > use them
>     >     >     > > > > in Virtualbox. If they run, they do not require
> anything
>     > else.
>     >     >     > > > >
>     >     >     > > > > On Wed, Oct 25, 2017 at 10:49 PM, Lotic Lists
>     >     >     > > > > <lists@lotic.com.br>
>     >     >     > > > wrote:
>     >     >     > > > >
>     >     >     > > > > > Sure but files on secondary storage don't have
>     > parents, I
>     >     >     > > > > > don't know if ACS can connect standalone files.
>     >     >     > > > > >
>     >     >     > > > > > vhd-util scan -f -m'*.vhd' -p
>     >     >     > > > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd
>     >     >     > > > > > capacity=311385128960
>     >     >     > > > > > size=304124744192 hidden=0 parent=none
>     >     >     > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd
>     >     >     > > > > > capacity=311385128960
>     >     >     > > > > > size=17770869248 hidden=0 parent=none
>     >     >     > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd
>     >     >     > > > > > capacity=311385128960
>     >     >     > > > > > size=52199817728 hidden=0 parent=none
>     >     >     > > > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd
>     >     >     > > > > > capacity=311385128960
>     >     >     > > > > > size=51615670784 hidden=0 parent=none
>     >     >     > > > > >
>     >     >     > > > > >
>     >     >     > > > > > -----Original Message-----
>     >     >     > > > > > From: Rafael Weingärtner [mailto:
> rafaelweingartner@
>     > gmail.com
>     >     > ]
>     >     >     > > > > > Sent: quarta-feira, 25 de outubro de 2017 22:35
>     >     >     > > > > > To: users@cloudstack.apache.org
>     >     >     > > > > > Subject: Re: Strange size of template from
> snapshot on
>     >     >     > > > > > XenServer
>     >     >     > > > > >
>     >     >     > > > > > If it was not executing coalesce you would see
> the
>     > parent
>     >     >     > > > > > reference when you listed the hierarchy, right?
>     >     >     > > > > >
>     >     >     > > > > > On Wed, Oct 25, 2017 at 10:25 PM, Lotic Lists
>     >     >     > > > > > <lists@lotic.com.br>
>     >     >     > > > > wrote:
>     >     >     > > > > >
>     >     >     > > > > > > I think ACS preserve all VHDs for coalesce but
>     > coalesce not
>     >     >     > > > > > > occur when create a template or before
> starting next
>     >     >     > > > > > > snapshot
>     >     >     > > scheduled.
>     >     >     > > > > > > If I create a template from any snapshots (2)
>     > showing in
>     >     >     > > > > > > GUI, the template have 49GB. 49 GB is the size
> of
>     > the 2
>     >     >     > > > > > > latest files on secondary
>     >     >     > > > > > storage.
>     >     >     > > > > > >
>     >     >     > > > > > > What do you think? Bug?
>     >     >     > > > > > >
>     >     >     > > > > > >
>     >     >     > > > > > > -----Original Message-----
>     >     >     > > > > > > From: Rafael Weingärtner
>     >     >     > > > > > > [mailto:rafaelweingartner@gmail.com]
>     >     >     > > > > > > Sent: quarta-feira, 25 de outubro de 2017 22:07
>     >     >     > > > > > > To: users@cloudstack.apache.org
>     >     >     > > > > > > Subject: Re: Strange size of template from
> snapshot
>     > on
>     >     >     > > > > > > XenServer
>     >     >     > > > > > >
>     >     >     > > > > > > Aha, The two destroyed entries are showing "
> removed:
>     >     > NULL".
>     >     >     > > > > > >
>     >     >     > > > > > > In ACS one thing is the "destroying" of a
> resource,
>     > another
>     >     >     > > > > > > is the removal of this resource from the
> system. I
>     > believe
>     >     >     > > > > > > there is something to do with the expunge
> interval,
>     > but I
>     >     > am
>     >     >     > > > > > > not sure if the snapshots complete removal also
>     > happens
>     >     >     > > > > > > within the expunge interval of user VMs or if
> it is
>     >     >     > > > > > > configured by something
>     >     >     > else.
>     >     >     > > > > > >
>     >     >     > > > > > >
>     >     >     > > > > > > On Wed, Oct 25, 2017 at 10:03 PM, Lotic Lists
>     >     >     > > > > > > <lists@lotic.com.br>
>     >     >     > > > > > wrote:
>     >     >     > > > > > >
>     >     >     > > > > > > > I not start manual snapshot, just scheduled
> with
>     > keep=2
>     >     >     > > > > > > >
>     >     >     > > > > > > > GUI showing 2 snapshots, database showing 4
>     > entries, 2
>     >     >     > > > > > > > Destroyed and
>     >     >     > > > > > > > 2 BackedUp
>     >     >     > > > > > > >
>     >     >     > > > > > > > *************************** 132. row
>     >     >     > ***************************
>     >     >     > > > > > > >               id: 6380
>     >     >     > > > > > > >   data_center_id: 2
>     >     >     > > > > > > >       account_id: 64
>     >     >     > > > > > > >        domain_id: 49
>     >     >     > > > > > > >        volume_id: 2102
>     >     >     > > > > > > > disk_offering_id: 230
>     >     >     > > > > > > >           status: Destroyed
>     >     >     > > > > > > >             path: NULL
>     >     >     > > > > > > >             name: FullCRM_ROOT-542_
> 20171022030205
>     >     >     > > > > > > >             uuid: 1294cdb7-01bf-4e76-b4c0-
>     > 11b3c6b395e9
>     >     >     > > > > > > >    snapshot_type: 4
>     >     >     > > > > > > > type_description: DAILY
>     >     >     > > > > > > >             size: 311385128960
>     >     >     > > > > > > >          created: 2017-10-22 03:02:05
>     >     >     > > > > > > >          removed: NULL
>     >     >     > > > > > > >   backup_snap_id: NULL
>     >     >     > > > > > > >         swift_id: NULL
>     >     >     > > > > > > >       sechost_id: NULL
>     >     >     > > > > > > >     prev_snap_id: NULL
>     >     >     > > > > > > >  hypervisor_type: XenServer
>     >     >     > > > > > > >          version: 2.2
>     >     >     > > > > > > >            s3_id: NULL
>     >     >     > > > > > > >         min_iops: NULL
>     >     >     > > > > > > >         max_iops: NULL
>     >     >     > > > > > > > *************************** 133. row
>     >     >     > ***************************
>     >     >     > > > > > > >               id: 6416
>     >     >     > > > > > > >   data_center_id: 2
>     >     >     > > > > > > >       account_id: 64
>     >     >     > > > > > > >        domain_id: 49
>     >     >     > > > > > > >        volume_id: 2102
>     >     >     > > > > > > > disk_offering_id: 230
>     >     >     > > > > > > >           status: Destroyed
>     >     >     > > > > > > >             path: NULL
>     >     >     > > > > > > >             name: FullCRM_ROOT-542_
> 20171023030205
>     >     >     > > > > > > >             uuid: 0a8067a3-ad15-40b6-8b16-
>     > c1881d7d9f41
>     >     >     > > > > > > >    snapshot_type: 4
>     >     >     > > > > > > > type_description: DAILY
>     >     >     > > > > > > >             size: 311385128960
>     >     >     > > > > > > >          created: 2017-10-23 03:02:05
>     >     >     > > > > > > >          removed: NULL
>     >     >     > > > > > > >   backup_snap_id: NULL
>     >     >     > > > > > > >         swift_id: NULL
>     >     >     > > > > > > >       sechost_id: NULL
>     >     >     > > > > > > >     prev_snap_id: NULL
>     >     >     > > > > > > >  hypervisor_type: XenServer
>     >     >     > > > > > > >          version: 2.2
>     >     >     > > > > > > >            s3_id: NULL
>     >     >     > > > > > > >         min_iops: NULL
>     >     >     > > > > > > >         max_iops: NULL
>     >     >     > > > > > > > *************************** 134. row
>     >     >     > ***************************
>     >     >     > > > > > > >               id: 6455
>     >     >     > > > > > > >   data_center_id: 2
>     >     >     > > > > > > >       account_id: 64
>     >     >     > > > > > > >        domain_id: 49
>     >     >     > > > > > > >        volume_id: 2102
>     >     >     > > > > > > > disk_offering_id: 230
>     >     >     > > > > > > >           status: BackedUp
>     >     >     > > > > > > >             path: NULL
>     >     >     > > > > > > >             name: FullCRM_ROOT-542_
> 20171024030207
>     >     >     > > > > > > >             uuid: 1ca646e5-a155-4a59-aa31-
>     > 5889eec7536f
>     >     >     > > > > > > >    snapshot_type: 4
>     >     >     > > > > > > > type_description: DAILY
>     >     >     > > > > > > >             size: 311385128960
>     >     >     > > > > > > >          created: 2017-10-24 03:02:07
>     >     >     > > > > > > >          removed: NULL
>     >     >     > > > > > > >   backup_snap_id: NULL
>     >     >     > > > > > > >         swift_id: NULL
>     >     >     > > > > > > >       sechost_id: NULL
>     >     >     > > > > > > >     prev_snap_id: NULL
>     >     >     > > > > > > >  hypervisor_type: XenServer
>     >     >     > > > > > > >          version: 2.2
>     >     >     > > > > > > >            s3_id: NULL
>     >     >     > > > > > > >         min_iops: NULL
>     >     >     > > > > > > >         max_iops: NULL
>     >     >     > > > > > > > *************************** 135. row
>     >     >     > ***************************
>     >     >     > > > > > > >               id: 6497
>     >     >     > > > > > > >   data_center_id: 2
>     >     >     > > > > > > >       account_id: 64
>     >     >     > > > > > > >        domain_id: 49
>     >     >     > > > > > > >        volume_id: 2102
>     >     >     > > > > > > > disk_offering_id: 230
>     >     >     > > > > > > >           status: BackedUp
>     >     >     > > > > > > >             path: NULL
>     >     >     > > > > > > >             name: FullCRM_ROOT-542_
> 20171025030208
>     >     >     > > > > > > >             uuid: 759d61dc-e3db-4815-b68b-
>     > b90904ccbda5
>     >     >     > > > > > > >    snapshot_type: 4
>     >     >     > > > > > > > type_description: DAILY
>     >     >     > > > > > > >             size: 311385128960
>     >     >     > > > > > > >          created: 2017-10-25 03:02:08
>     >     >     > > > > > > >          removed: NULL
>     >     >     > > > > > > >   backup_snap_id: NULL
>     >     >     > > > > > > >         swift_id: NULL
>     >     >     > > > > > > >       sechost_id: NULL
>     >     >     > > > > > > >     prev_snap_id: NULL
>     >     >     > > > > > > >  hypervisor_type: XenServer
>     >     >     > > > > > > >          version: 2.2
>     >     >     > > > > > > >            s3_id: NULL
>     >     >     > > > > > > >         min_iops: NULL
>     >     >     > > > > > > >         max_iops: NULL
>     >     >     > > > > > > >
>     >     >     > > > > > > >
>     >     >     > > > > > > >
>     >     >     > > > > > > > -----Original Message-----
>     >     >     > > > > > > > From: Rafael Weingärtner
>     >     >     > > > > > > > [mailto:rafaelweingartner@gmail.com]
>     >     >     > > > > > > > Sent: quarta-feira, 25 de outubro de 2017
> 21:50
>     >     >     > > > > > > > To: users@cloudstack.apache.org
>     >     >     > > > > > > > Subject: Re: Strange size of template from
>     > snapshot on
>     >     >     > > > > > > > XenServer
>     >     >     > > > > > > >
>     >     >     > > > > > > > The file "14818597-55bb-49be-9ace-
>     > 761e9e01c074.vhd"
>     >     > seems
>     >     >     > > > > > > > more like a manual snapshot. It was created 2
>     > hours after
>     >     >     > > > > > > > the
>     >     >     > "normal"
>     >     >     > > > > > > > time. I would check these snapshots in ACS
> and see
>     > how
>     >     >     > > > > > > > they are
>     >     >     > > > > presented.
>     >     >     > > > > > > > BTW, how many snapshots is ACS showing for
> this
>     > volume? I
>     >     >     > > > > > > > would also check the "snapshots" table for
> the
>     > volume you
>     >     >     > > > > > > > have configured
>     >     >     > > > > them.
>     >     >     > > > > > > > select * from snapshots where volume_id = ?
>     >     >     > > > > > > >
>     >     >     > > > > > > > you may also filter for removed:
>     >     >     > > > > > > > select * from snapshots where volume_id = ?
> and
>     > removed
>     >     > is
>     >     >     > > > > > > > null
>     >     >     > > > > > > >
>     >     >     > > > > > > > On Wed, Oct 25, 2017 at 9:39 PM, Lotic Lists
>     >     >     > > > > > > > <lists@lotic.com.br>
>     >     >     > > > > > wrote:
>     >     >     > > > > > > >
>     >     >     > > > > > > > > Hi Rafael, I check
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > # ls -lrt
>     >     >     > > > > > > > > total 285368382
>     >     >     > > > > > > > > -rw-r--r-- 1 root root 304124744192 Oct 22
> 03:28
>     >     >     > > > > > > > > 14818597-55bb-49be-9ace- 761e9e01c074.vhd
>     >     >     > > > > > > > > -rw-r--r-- 1 root root  17770869248 Oct 23
> 01:10
>     >     >     > > > > > > > > bbd9e9b5-53fb-44a2-afbe- f262396a2d84.vhd
>     >     >     > > > > > > > > -rw-r--r-- 1 root root  52199817728 Oct 24
> 01:30
>     >     >     > > > > > > > > 3b90bbb2-7ce5-41f6-9f7e- fd0ea061cb2d.vhd
>     >     >     > > > > > > > > -rw-r--r-- 1 root root  51615670784 Oct 25
> 01:28
>     >     >     > > > > > > > > 4b450217-c9ad-45a2-946b- de3cb323469b.vhd
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > # vhd-util scan -f -m'*.vhd' -p
>     >     >     > > > > > > > > vhd=14818597-55bb-49be-9ace-
> 761e9e01c074.vhd
>     >     >     > > > > > > > > capacity=311385128960
>     >     >     > > > > > > > > size=304124744192 hidden=0 parent=none
>     >     >     > > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-
> f262396a2d84.vhd
>     >     >     > > > > > > > > capacity=311385128960
>     >     >     > > > > > > > > size=17770869248 hidden=0 parent=none
>     >     >     > > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-
> fd0ea061cb2d.vhd
>     >     >     > > > > > > > > capacity=311385128960
>     >     >     > > > > > > > > size=52199817728 hidden=0 parent=none
>     >     >     > > > > > > > > vhd=4b450217-c9ad-45a2-946b-
> de3cb323469b.vhd
>     >     >     > > > > > > > > capacity=311385128960
>     >     >     > > > > > > > > size=51615670784 hidden=0 parent=none
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > -----Original Message-----
>     >     >     > > > > > > > > From: Rafael Weingärtner
>     >     >     > > > > > > > > [mailto:rafaelweingartner@gmail.com]
>     >     >     > > > > > > > > Sent: quarta-feira, 25 de outubro de 2017
> 20:44
>     >     >     > > > > > > > > To: users@cloudstack.apache.org
>     >     >     > > > > > > > > Subject: Re: Strange size of template from
>     > snapshot on
>     >     >     > > > > > > > > XenServer
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > Did you check the hierarchy of those
> snapshot
>     > files in
>     >     >     > > > > > > > > your secondary storage?
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > On Wed, Oct 25, 2017 at 7:44 PM, Lotic
> Lists
>     >     >     > > > > > > > > <lists@lotic.com.br>
>     >     >     > > > > > > wrote:
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > > Hi all
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > Anyone identify problems with templates
> from
>     >     > snapshots
>     >     >     > > > > > > > > > on
>     >     >     > > > > > XenServer?
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > I created a recurrent snapshot, first
> VHD on
>     >     > secondary
>     >     >     > > > > > > > > > storage have a similar size of volume.
> If I
>     > create a
>     >     >     > > > > > > > > > template from latest snapshot, the size
> of VHD
>     > of
>     >     >     > > > > > > > > > template is smaller equal from last
> snapshot,
>     > I think
>     >     >     > > > > > > > > > CloudStack/Xenserver not coalesce VHD
> files of
>     >     >     > > > > > > > > > secondary storage for create the template
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > I keep two recurrent snapshots, on
> secondary
>     > storage
>     >     >     > > > > > > > > > have four
>     >     >     > > > > > files.
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > Environment is ACS 4.9.3.0 and XenServer
> 6.5,
>     > primary
>     >     >     > > > > > > > > > storage is PreSetup with ISCSI.
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > Thanks
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > > Marcelo
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > > > --
>     >     >     > > > > > > > > Rafael Weingärtner
>     >     >     > > > > > > > >
>     >     >     > > > > > > > >
>     >     >     > > > > > > >
>     >     >     > > > > > > >
>     >     >     > > > > > > > --
>     >     >     > > > > > > > Rafael Weingärtner
>     >     >     > > > > > > >
>     >     >     > > > > > > >
>     >     >     > > > > > >
>     >     >     > > > > > >
>     >     >     > > > > > > --
>     >     >     > > > > > > Rafael Weingärtner
>     >     >     > > > > > >
>     >     >     > > > > > >
>     >     >     > > > > >
>     >     >     > > > > >
>     >     >     > > > > > --
>     >     >     > > > > > Rafael Weingärtner
>     >     >     > > > > >
>     >     >     > > > > >
>     >     >     > > > >
>     >     >     > > > >
>     >     >     > > > > --
>     >     >     > > > > Rafael Weingärtner
>     >     >     > > > >
>     >     >     > > > >
>     >     >     > > > >
>     >     >     > > >
>     >     >     > > >
>     >     >     > > > --
>     >     >     > > > Rafael Weingärtner
>     >     >     > > >
>     >     >     > > >
>     >     >     > >
>     >     >     > >
>     >     >     > > --
>     >     >     > > Rafael Weingärtner
>     >     >     > >
>     >     >     > >
>     >     >     >
>     >     >     >
>     >     >     > --
>     >     >     > Rafael Weingärtner
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >     --
>     >     >     Rafael Weingärtner
>     >     >
>     >     >
>     >     >
>     >     > DISCLAIMER
>     >     > ==========
>     >     > This e-mail may contain privileged and confidential information
>     > which is
>     >     > the property of Accelerite, a Persistent Systems business. It
> is
>     > intended
>     >     > only for the use of the individual or entity to which it is
>     > addressed. If
>     >     > you are not the intended recipient, you are not authorized to
> read,
>     > retain,
>     >     > copy, print, distribute or use this message. If you have
> received
>     > this
>     >     > communication in error, please notify the sender and delete all
>     > copies of
>     >     > this message. Accelerite, a Persistent Systems business does
> not
>     > accept any
>     >     > liability for virus infected mails.
>     >     >
>     >
>     >
>     >
>     >     --
>     >     Rafael Weingärtner
>     >
>     >
>     > DISCLAIMER
>     > ==========
>     > This e-mail may contain privileged and confidential information
> which is
>     > the property of Accelerite, a Persistent Systems business. It is
> intended
>     > only for the use of the individual or entity to which it is
> addressed. If
>     > you are not the intended recipient, you are not authorized to read,
> retain,
>     > copy, print, distribute or use this message. If you have received
> this
>     > communication in error, please notify the sender and delete all
> copies of
>     > this message. Accelerite, a Persistent Systems business does not
> accept any
>     > liability for virus infected mails.
>     >
>
>
>
>     --
>     Rafael Weingärtner
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message