cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francois Gaudreault <fgaudrea...@cloudops.com>
Subject Re: S3/Swift Problem around Virtual Size
Date Tue, 26 Aug 2014 10:30:13 GMT
I mean we are populating the template just like we would do with normal NFS
using the UI.

ACS takes care of pushing to swift.

FG
On Aug 26, 2014 6:02 AM, "Francois Gaudreault" <fgaudreault@cloudops.com>
wrote:

> Punith,
>
> We are using Swift. We have a tmpauth proxy.
>
> FG
> On Aug 26, 2014 2:48 AM, "Punith S" <punith.s@cloudbyte.com> wrote:
>
>> sure mike,
>>
>> since i don't have a S3 account, i'm getting one today.
>>
>> francois, can you brief me how you seeded your templates into S3.
>>
>> thanks!
>>
>>
>> On Mon, Aug 25, 2014 at 11:16 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com> wrote:
>>
>>> Yes, I expect we'll see the same issue with S3, as well.
>>>
>>> Punith - Is this something you might have time to investigate? Perhaps
>>> Edison can point us in the right direction here.
>>>
>>>
>>> On Mon, Aug 25, 2014 at 5:17 AM, Francois Gaudreault <
>>> fgaudreault@cloudops.com> wrote:
>>>
>>> > Punith,
>>> >
>>> > I highly anticipate the same issue with S3... it shares a lot of code
>>> with
>>> > swift.
>>> >
>>> > My focus would be swift, but we should fix for both :)
>>> >
>>> > FG
>>> > On Aug 25, 2014 6:33 AM, "Punith S" <punith.s@cloudbyte.com> wrote:
>>> >
>>> > > thanks for opening this thread mike,
>>> > >
>>> > > since i only use nfs as my secondary storage provider, i didn't see
>>> this
>>> > > issue till date.
>>> > >
>>> > > is this issue occurring even using a S3 secondary storage with
>>> staging
>>> > nfs
>>> > > store ?
>>> > >
>>> > > if so like edison pointed we need to fetch the virtual size from the
>>> nfs
>>> > > store instead of S3 in the deploymentmanager.
>>> > >
>>> > > thanks
>>> > >
>>> > >
>>> > > On Sat, Aug 23, 2014 at 3:45 AM, Mike Tutkowski <
>>> > > mike.tutkowski@solidfire.com> wrote:
>>> > >
>>> > > > Hey Edison,
>>> > > >
>>> > > > Do you know how difficult/easy of a fix this is, who might be
>>> available
>>> > > to
>>> > > > put this fix in, and for what release (hopefully 4.4.1) this fix
>>> could
>>> > > find
>>> > > > its way in?
>>> > > >
>>> > > > Thanks!
>>> > > > Mike
>>> > > >
>>> > > >
>>> > > > On Fri, Aug 22, 2014 at 3:37 PM, Francois Gaudreault <
>>> > > > fgaudreault@cloudops.com> wrote:
>>> > > >
>>> > > > > Min,
>>> > > > >
>>> > > > > Ok, but this is not the behavior I see. Even without requesting
>>> a VM
>>> > > > > create, the template is pushed to the staging NFS at least
once.
>>> Is
>>> > it
>>> > > > > downloaded there or pushed after download, that I am not
sure. I
>>> was
>>> > > > > assuming the swift upload bash script was executed after
the
>>> template
>>> > > is
>>> > > > on
>>> > > > > the staging.
>>> > > > >
>>> > > > > Anyway... the focus is on the virt size, and you all know
the
>>> code
>>> > > better
>>> > > > > than I do :)
>>> > > > >
>>> > > > > FG
>>> > > > > On Aug 22, 2014 5:28 PM, "Min Chen" <min.chen@citrix.com>
wrote:
>>> > > > >
>>> > > > >> No. For S3/Swift, register template will directly upload
>>> templates
>>> > to
>>> > > S3
>>> > > > >> without going through staging NFS. It will only be copied
to
>>> staging
>>> > > NFS
>>> > > > >> when we first use that template to provision a VM.
>>> > > > >>
>>> > > > >> Thanks
>>> > > > >> -min
>>> > > > >>
>>> > > > >> On 8/22/14 2:25 PM, "Francois Gaudreault" <
>>> fgaudreault@cloudops.com
>>> > >
>>> > > > >> wrote:
>>> > > > >>
>>> > > > >> >Edison,
>>> > > > >> >
>>> > > > >> >Isnt the templates downloaded to the Staging NFS
first?
>>> > > > >> >
>>> > > > >> >FG
>>> > > > >> >On Aug 22, 2014 5:20 PM, "Edison Su" <Edison.su@citrix.com>
>>> wrote:
>>> > > > >> >
>>> > > > >> >> I know the reason why the size of template doesn¹t
have
>>> correct
>>> > > > virtual
>>> > > > >> >> size if it¹s registered in S3/Swift:
>>> > > > >> >> In case of s3/swift, the template is directly
stored into
>>> > s3/swift
>>> > > > >> >>through
>>> > > > >> >> swift/s3 api, there is no place for cloudstack
to look into
>>> > > template,
>>> > > > >> to
>>> > > > >> >> find out the virtual size during template registration.
>>> > > > >> >> While, if secondary storage is NFS, the template
is first
>>> stored
>>> > on
>>> > > > >> >> NFS(which has file system), cloudstack can unzip
the
>>> template(if
>>> > > > it¹s a
>>> > > > >> >> zipped file), and read virtual size from the
file, then
>>> report
>>> > back
>>> > > > to
>>> > > > >> >>mgt
>>> > > > >> >> server.
>>> > > > >> >> In order to fix it, we can add some code as:
all the
>>> templates
>>> > > > >> >>registered
>>> > > > >> >> on Swift/S3, need to be downloaded to a NFS
intermediate
>>> storage
>>> > > > before
>>> > > > >> >>it
>>> > > > >> >> can be consumed by primary storage. After the
download
>>> finished,
>>> > > then
>>> > > > >> we
>>> > > > >> >> check virtual size of template, and report back
to mgt
>>> > > server/update
>>> > > > DB
>>> > > > >> >>etc.
>>> > > > >> >>
>>> > > > >> >> From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
>>> > > > >> >> Sent: Friday, August 22, 2014 1:38 PM
>>> > > > >> >> To: dev@cloudstack.apache.org
>>> > > > >> >> Cc: Edison Su
>>> > > > >> >> Subject: S3/Swift Problem around Virtual Size
>>> > > > >> >>
>>> > > > >> >> Hi,
>>> > > > >> >>
>>> > > > >> >> This was brought up in a different e-mail thread,
but I
>>> wanted to
>>> > > > make
>>> > > > >> >>it
>>> > > > >> >> more clear that it's related to CloudStack's
download code
>>> around
>>> > > > >> >>S3/Swift,
>>> > > > >> >> so I'm opening up a new thread.
>>> > > > >> >>
>>> > > > >> >> Francois (from CloudOps) noticed today that
when he
>>> downloaded a
>>> > > > >> >>template
>>> > > > >> >> (VHD format) to Swift (but it looks like the
same applies
>>> for S3)
>>> > > > that
>>> > > > >> >>the
>>> > > > >> >> physical and virtual sizes are set to be the
same.
>>> > > > >> >>
>>> > > > >> >> This appears to have the following consequence:
>>> > > > >> >>
>>> > > > >> >> You can download a template with a physical
size of, say, 3
>>> GB
>>> > and
>>> > > a
>>> > > > >> >>root
>>> > > > >> >> disk that's supposed to be, say, 20 GB. Instead
of the
>>> virtual
>>> > size
>>> > > > >> >>showing
>>> > > > >> >> as 20 GB, it shows as 3 GB.
>>> > > > >> >>
>>> > > > >> >> This is not an issue with NFS. In that situation,
the two
>>> sizes
>>> > are
>>> > > > >> >> correctly accounted for.
>>> > > > >> >>
>>> > > > >> >> What later can happen is the template is downloaded
from
>>> Swift
>>> > and
>>> > > > >> takes
>>> > > > >> >> up an unexpected amount of space on the XenServer
storage
>>> > > repository
>>> > > > >> >>(SR).
>>> > > > >> >>
>>> > > > >> >> If there is enough space on the SR, this isn't
too big of a
>>> deal.
>>> > > > >> >>However,
>>> > > > >> >> for so-called managed storage plug-ins (examples
are
>>> SolidFire
>>> > and
>>> > > > >> >> CloudByte), this will lead to them dynamically
creating a SAN
>>> > > volume
>>> > > > of
>>> > > > >> >>the
>>> > > > >> >> wrong size.
>>> > > > >> >>
>>> > > > >> >> Francois opened up the following ticket:
>>> > > > >> >>
>>> > > > >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406
>>> > > > >> >>
>>> > > > >> >> Thanks!
>>> > > > >> >>
>>> > > > >> >> --
>>> > > > >> >> Mike Tutkowski
>>> > > > >> >> Senior CloudStack Developer, SolidFire Inc.
>>> > > > >> >> e: mike.tutkowski@solidfire.com<mailto:
>>> > > mike.tutkowski@solidfire.com>
>>> > > > >> >> o: 303.746.7302
>>> > > > >> >> Advancing the way the world uses the cloud<
>>> > > > >> >> http://solidfire.com/solution/overview/?video=play>
>>> > > > >> >>
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > > >
>>> > > > --
>>> > > > *Mike Tutkowski*
>>> > > > *Senior CloudStack Developer, SolidFire Inc.*
>>> > > > e: mike.tutkowski@solidfire.com
>>> > > > o: 303.746.7302
>>> > > > Advancing the way the world uses the cloud
>>> > > > <http://solidfire.com/solution/overview/?video=play>*™*
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > regards,
>>> > >
>>> > > punith s
>>> > > cloudbyte.com
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>>
>>
>> --
>> regards,
>>
>> punith s
>> cloudbyte.com
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message