cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francois Gaudreault <fgaudrea...@cloudops.com>
Subject Re: S3/Swift Problem around Virtual Size
Date Tue, 26 Aug 2014 10:02:43 GMT
Punith,

We are using Swift. We have a tmpauth proxy.

FG
On Aug 26, 2014 2:48 AM, "Punith S" <punith.s@cloudbyte.com> wrote:

> sure mike,
>
> since i don't have a S3 account, i'm getting one today.
>
> francois, can you brief me how you seeded your templates into S3.
>
> thanks!
>
>
> On Mon, Aug 25, 2014 at 11:16 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> Yes, I expect we'll see the same issue with S3, as well.
>>
>> Punith - Is this something you might have time to investigate? Perhaps
>> Edison can point us in the right direction here.
>>
>>
>> On Mon, Aug 25, 2014 at 5:17 AM, Francois Gaudreault <
>> fgaudreault@cloudops.com> wrote:
>>
>> > Punith,
>> >
>> > I highly anticipate the same issue with S3... it shares a lot of code
>> with
>> > swift.
>> >
>> > My focus would be swift, but we should fix for both :)
>> >
>> > FG
>> > On Aug 25, 2014 6:33 AM, "Punith S" <punith.s@cloudbyte.com> wrote:
>> >
>> > > thanks for opening this thread mike,
>> > >
>> > > since i only use nfs as my secondary storage provider, i didn't see
>> this
>> > > issue till date.
>> > >
>> > > is this issue occurring even using a S3 secondary storage with staging
>> > nfs
>> > > store ?
>> > >
>> > > if so like edison pointed we need to fetch the virtual size from the
>> nfs
>> > > store instead of S3 in the deploymentmanager.
>> > >
>> > > thanks
>> > >
>> > >
>> > > On Sat, Aug 23, 2014 at 3:45 AM, Mike Tutkowski <
>> > > mike.tutkowski@solidfire.com> wrote:
>> > >
>> > > > Hey Edison,
>> > > >
>> > > > Do you know how difficult/easy of a fix this is, who might be
>> available
>> > > to
>> > > > put this fix in, and for what release (hopefully 4.4.1) this fix
>> could
>> > > find
>> > > > its way in?
>> > > >
>> > > > Thanks!
>> > > > Mike
>> > > >
>> > > >
>> > > > On Fri, Aug 22, 2014 at 3:37 PM, Francois Gaudreault <
>> > > > fgaudreault@cloudops.com> wrote:
>> > > >
>> > > > > Min,
>> > > > >
>> > > > > Ok, but this is not the behavior I see. Even without requesting
a
>> VM
>> > > > > create, the template is pushed to the staging NFS at least once.
>> Is
>> > it
>> > > > > downloaded there or pushed after download, that I am not sure.
I
>> was
>> > > > > assuming the swift upload bash script was executed after the
>> template
>> > > is
>> > > > on
>> > > > > the staging.
>> > > > >
>> > > > > Anyway... the focus is on the virt size, and you all know the
code
>> > > better
>> > > > > than I do :)
>> > > > >
>> > > > > FG
>> > > > > On Aug 22, 2014 5:28 PM, "Min Chen" <min.chen@citrix.com>
wrote:
>> > > > >
>> > > > >> No. For S3/Swift, register template will directly upload
>> templates
>> > to
>> > > S3
>> > > > >> without going through staging NFS. It will only be copied
to
>> staging
>> > > NFS
>> > > > >> when we first use that template to provision a VM.
>> > > > >>
>> > > > >> Thanks
>> > > > >> -min
>> > > > >>
>> > > > >> On 8/22/14 2:25 PM, "Francois Gaudreault" <
>> fgaudreault@cloudops.com
>> > >
>> > > > >> wrote:
>> > > > >>
>> > > > >> >Edison,
>> > > > >> >
>> > > > >> >Isnt the templates downloaded to the Staging NFS first?
>> > > > >> >
>> > > > >> >FG
>> > > > >> >On Aug 22, 2014 5:20 PM, "Edison Su" <Edison.su@citrix.com>
>> wrote:
>> > > > >> >
>> > > > >> >> I know the reason why the size of template doesn¹t
have
>> correct
>> > > > virtual
>> > > > >> >> size if it¹s registered in S3/Swift:
>> > > > >> >> In case of s3/swift, the template is directly stored
into
>> > s3/swift
>> > > > >> >>through
>> > > > >> >> swift/s3 api, there is no place for cloudstack to
look into
>> > > template,
>> > > > >> to
>> > > > >> >> find out the virtual size during template registration.
>> > > > >> >> While, if secondary storage is NFS, the template
is first
>> stored
>> > on
>> > > > >> >> NFS(which has file system), cloudstack can unzip
the
>> template(if
>> > > > it¹s a
>> > > > >> >> zipped file), and read virtual size from the file,
then report
>> > back
>> > > > to
>> > > > >> >>mgt
>> > > > >> >> server.
>> > > > >> >> In order to fix it, we can add some code as: all
the templates
>> > > > >> >>registered
>> > > > >> >> on Swift/S3, need to be downloaded to a NFS intermediate
>> storage
>> > > > before
>> > > > >> >>it
>> > > > >> >> can be consumed by primary storage. After the download
>> finished,
>> > > then
>> > > > >> we
>> > > > >> >> check virtual size of template, and report back
to mgt
>> > > server/update
>> > > > DB
>> > > > >> >>etc.
>> > > > >> >>
>> > > > >> >> From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
>> > > > >> >> Sent: Friday, August 22, 2014 1:38 PM
>> > > > >> >> To: dev@cloudstack.apache.org
>> > > > >> >> Cc: Edison Su
>> > > > >> >> Subject: S3/Swift Problem around Virtual Size
>> > > > >> >>
>> > > > >> >> Hi,
>> > > > >> >>
>> > > > >> >> This was brought up in a different e-mail thread,
but I
>> wanted to
>> > > > make
>> > > > >> >>it
>> > > > >> >> more clear that it's related to CloudStack's download
code
>> around
>> > > > >> >>S3/Swift,
>> > > > >> >> so I'm opening up a new thread.
>> > > > >> >>
>> > > > >> >> Francois (from CloudOps) noticed today that when
he
>> downloaded a
>> > > > >> >>template
>> > > > >> >> (VHD format) to Swift (but it looks like the same
applies for
>> S3)
>> > > > that
>> > > > >> >>the
>> > > > >> >> physical and virtual sizes are set to be the same.
>> > > > >> >>
>> > > > >> >> This appears to have the following consequence:
>> > > > >> >>
>> > > > >> >> You can download a template with a physical size
of, say, 3 GB
>> > and
>> > > a
>> > > > >> >>root
>> > > > >> >> disk that's supposed to be, say, 20 GB. Instead
of the virtual
>> > size
>> > > > >> >>showing
>> > > > >> >> as 20 GB, it shows as 3 GB.
>> > > > >> >>
>> > > > >> >> This is not an issue with NFS. In that situation,
the two
>> sizes
>> > are
>> > > > >> >> correctly accounted for.
>> > > > >> >>
>> > > > >> >> What later can happen is the template is downloaded
from Swift
>> > and
>> > > > >> takes
>> > > > >> >> up an unexpected amount of space on the XenServer
storage
>> > > repository
>> > > > >> >>(SR).
>> > > > >> >>
>> > > > >> >> If there is enough space on the SR, this isn't too
big of a
>> deal.
>> > > > >> >>However,
>> > > > >> >> for so-called managed storage plug-ins (examples
are SolidFire
>> > and
>> > > > >> >> CloudByte), this will lead to them dynamically creating
a SAN
>> > > volume
>> > > > of
>> > > > >> >>the
>> > > > >> >> wrong size.
>> > > > >> >>
>> > > > >> >> Francois opened up the following ticket:
>> > > > >> >>
>> > > > >> >> https://issues.apache.org/jira/browse/CLOUDSTACK-7406
>> > > > >> >>
>> > > > >> >> Thanks!
>> > > > >> >>
>> > > > >> >> --
>> > > > >> >> Mike Tutkowski
>> > > > >> >> Senior CloudStack Developer, SolidFire Inc.
>> > > > >> >> e: mike.tutkowski@solidfire.com<mailto:
>> > > mike.tutkowski@solidfire.com>
>> > > > >> >> o: 303.746.7302
>> > > > >> >> Advancing the way the world uses the cloud<
>> > > > >> >> http://solidfire.com/solution/overview/?video=play>
>> > > > >> >>
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> > > > --
>> > > > *Mike Tutkowski*
>> > > > *Senior CloudStack Developer, SolidFire Inc.*
>> > > > e: mike.tutkowski@solidfire.com
>> > > > o: 303.746.7302
>> > > > Advancing the way the world uses the cloud
>> > > > <http://solidfire.com/solution/overview/?video=play>*™*
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > regards,
>> > >
>> > > punith s
>> > > cloudbyte.com
>> > >
>> >
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>
>
>
> --
> regards,
>
> punith s
> cloudbyte.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message