cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus <shadow...@gmail.com>
Subject Re: [PROPOSAL] ROOT volume detach - feature for CS 4.5
Date Fri, 28 Mar 2014 23:30:09 GMT
Yeah, VMware has many formats for storage and configs, and some suffer
from insufficient abstraction between resources and definitions for
cloud use. I'm sure it can be worked around in some regard.

I still haven't wrapped my head around the datastore(or SR for
xen)-per-volume, but I know that's how the SolidFire plugin works, so
forgive me if I've made assumptions that make sense in the scope of
how our plugins work.

On Fri, Mar 28, 2014 at 5:22 PM, Mike Tutkowski
<mike.tutkowski@solidfire.com> wrote:
> See...I don't know the history behind this, but for VMware, when we shut a
> VM down, the config files remain, which is not how this works on XenServer
> or KVM.
>
> That is the "root" (pun intended) of our managed-storage problem here.
>
>
> On Fri, Mar 28, 2014 at 5:20 PM, Mike Tutkowski
> <mike.tutkowski@solidfire.com> wrote:
>>
>> Sure :) In the storage_pool table, there is a column called "managed". 1 =
>> managed
>>
>>
>> On Fri, Mar 28, 2014 at 5:18 PM, Alena Prokharchyk
>> <Alena.Prokharchyk@citrix.com> wrote:
>>>
>>> Ok, then can you please tell me the way how to determine if the
>>> corresponding storage is managed, by looking at CS DB entry?
>>>
>>>  For phase #1 of the feature, I will just implement it for the regular
>>> storage in KVM/Xen/VmWare; and implement managed storage support some time
>>> later.
>>>
>>> -Alena.
>>>
>>> From: Mike Tutkowski <mike.tutkowski@solidfire.com>
>>> Date: Friday, March 28, 2014 at 4:15 PM
>>>
>>> To: Alena Prokharchyk <alena.prokharchyk@citrix.com>
>>> Cc: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>,
>>> "shadowsor@gmail.com" <shadowsor@gmail.com>
>>> Subject: Re: [PROPOSAL] ROOT volume detach - feature for CS 4.5
>>>
>>> Yes
>>>
>>> With non-managed storage, the admin determines when to manually create
>>> and delete datastores.
>>>
>>> I think this will only be a problem with managed storage on VMware.
>>>
>>>
>>> On Fri, Mar 28, 2014 at 5:14 PM, Alena Prokharchyk
>>> <Alena.Prokharchyk@citrix.com> wrote:
>>>>
>>>> So it only affects managed storage?
>>>>
>>>> From: Mike Tutkowski <mike.tutkowski@solidfire.com>
>>>> Date: Friday, March 28, 2014 at 4:10 PM
>>>> To: Alena Prokharchyk <alena.prokharchyk@citrix.com>
>>>> Cc: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>,
>>>> "shadowsor@gmail.com" <shadowsor@gmail.com>
>>>>
>>>> Subject: Re: [PROPOSAL] ROOT volume detach - feature for CS 4.5
>>>>
>>>> Let me illustrate this with an example:
>>>>
>>>> * User creates a VM whose root disk is placed on managed storage
>>>>
>>>> * Storage plug-in creates a volume on its SAN
>>>>
>>>> * VMware server resource creates a datastore based on the newly created
>>>> SAN volume (let me stress that this datastore was created by the VMware
>>>> server resource - not manually by an admin as would be the case for
>>>> non-managed storage)
>>>>
>>>> * Inside the datastore are placed the VMDK file (root disk) along with
>>>> VM config files like VMX, NVRAM, etc.
>>>>
>>>> * User detaches the root volume (the VMDK file and VM config files
>>>> remain in the datastore)
>>>>
>>>> * User attaches another root volume to the VM (the VMDK file is stored
>>>> in a datastore different from the datastore where the VM config files
>>>> reside, which is fine for now)
>>>>
>>>> * User deletes and expunges the original root disk (this leads to the
>>>> datastore the VMDK file is on being removed...as a side effect, you will
>>>> also lose your VM config files), SAN volume is deleted, CloudStack volume
is
>>>> marked as deleted in the database
>>>>
>>>>
>>>> On Fri, Mar 28, 2014 at 5:05 PM, Mike Tutkowski
>>>> <mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>> So, do you guys see my concern with VMware, though?
>>>>>
>>>>> VMware is different from XenServer and KVM in that its VM config files
>>>>> are stored in the datastore along side the root volume (in CloudStack
4.3,
>>>>> for example).
>>>>>
>>>>> If you switch the VM to use a VMDK file in a different datastore, the
>>>>> config files will remain in the original datastore (unless we transfer
them
>>>>> ourselves to the new datastore).
>>>>>
>>>>> If they remain in the original datastore and that disk is deleted
>>>>> later, the datastore that contains that disk will be removed (along with
the
>>>>> VM config files that are new being used in conjunction with a disk in
>>>>> another datastore).
>>>>>
>>>>>
>>>>> On Fri, Mar 28, 2014 at 4:58 PM, Alena Prokharchyk
>>>>> <Alena.Prokharchyk@citrix.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 3/28/14, 3:50 PM, "Marcus" <shadowsor@gmail.com> wrote:
>>>>>>
>>>>>> > I see this feature as mainly just shuffling around object properties
>>>>>> >in the database. I don't expect any major issues to arise with
any
>>>>>> >storage if an inactive "root" disk is marked as a "data" disk
in the
>>>>>> >DB, for example. In the end, when you start a VM you're always
going
>>>>>> >to have a root disk in the vm instance object, and volumes that
are
>>>>>> >attached/detached are going to be passed as data disks (If I
>>>>>> >understand correctly). It doesn't really matter to the storage
>>>>>> > drivers
>>>>>> >if the volume object was previously of type root or data.
>>>>>>
>>>>>> Correct. That¹s what I reflected in the spec. But I¹m going to
test it
>>>>>> on
>>>>>> all major supported hypervisors - KVM/Xen/VmWare - anyway, just to
be
>>>>>> 100%
>>>>>> sure nothing breaks.
>>>>>>
>>>>>>
>>>>>>
>>>>>> >
>>>>>> >On Fri, Mar 28, 2014 at 12:48 PM, Alena Prokharchyk
>>>>>> ><Alena.Prokharchyk@citrix.com> wrote:
>>>>>> >> I will look into it more, Mike. vmWare indeed can be different.
>>>>>> >>
>>>>>> >> -Alena.
>>>>>> >>
>>>>>> >> From: Mike Tutkowski
>>>>>> >><mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>>
>>>>>> >> Date: Friday, March 28, 2014 at 11:39 AM
>>>>>> >> To: Alena Prokharchyk
>>>>>> >><alena.prokharchyk@citrix.com<mailto:alena.prokharchyk@citrix.com>>
>>>>>> >> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>>>>>> >><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>>>>>> >> Subject: Re: [PROPOSAL] ROOT volume detach - feature for
CS 4.5
>>>>>> >>
>>>>>> >> VMware is also different because when you shut a VMware
VM down
>>>>>> >> from
>>>>>> >>CloudStack, the VM still exists in vCenter Server (whereas
for
>>>>>> >> XenServer
>>>>>> >>and KVM, the VM is gone).
>>>>>> >>
>>>>>> >> Since the life of a datastore that was created for managed
storage
>>>>>> >> is
>>>>>> >>tied to the life of the CloudStack volume it stores, when
the
>>>>>> >> CloudStack
>>>>>> >>volume is deleted, the datastore goes away, as well.
>>>>>> >>
>>>>>> >> If the datastore in question was automatically created to
store a
>>>>>> >> root
>>>>>> >>disk (alongside VM config files) and you switch the VM to
another
>>>>>> >> root
>>>>>> >>disk (which has to necessarily be in another datastore),
you won't
>>>>>> >> see a
>>>>>> >>problem until the original root volume is expunged by CloudStack.
At
>>>>>> >>this point, its datastore will go away along with your VM
config
>>>>>> >> files.
>>>>>> >>
>>>>>> >>
>>>>>> >> On Fri, Mar 28, 2014 at 12:31 PM, Mike Tutkowski
>>>>>> >><mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>>
>>>>>> >>wrote:
>>>>>> >> Well, the reason I brought it up was mainly due to VMware.
>>>>>> >>
>>>>>> >> Let's use that as an example:
>>>>>> >>
>>>>>> >> I initiate the process of spinning up a VM based on managed
>>>>>> >> storage.
>>>>>> >> A volume is dynamically created on a SAN.
>>>>>> >> VmwareStorageProcessor dynamically creates a datastore to
consume
>>>>>> >> the
>>>>>> >>newly created SAN volume.
>>>>>> >> All VMware VM files (ex. VMX, NVRAM) are placed in the datastore
>>>>>> >>alongside the VMDK file that represents the root volume.
>>>>>> >>
>>>>>> >> Now, let's say we want to detach this root volume and give
the VM a
>>>>>> >> new
>>>>>> >>root volume.
>>>>>> >>
>>>>>> >> The new root volume will necessarily be on a different datastore
>>>>>> >> than
>>>>>> >>the datastore of the previous root volume (because a datastore
>>>>>> >> created
>>>>>> >>to consume managed storage will have at most one VMDK file*).
>>>>>> >>
>>>>>> >> Is it going to be a problem that the VM's files (ex. VMX,
NVRAM)
>>>>>> >> are on
>>>>>> >>one datastore, but its root disk is on another?
>>>>>> >>
>>>>>> >> I don't think it's really a problem until you go to delete
the
>>>>>> >> original
>>>>>> >>root volume from CloudStack. At that point, its datastore
will be
>>>>>> >>removed (including, of course, your VM's VMX, NVRAM, etc.
files).
>>>>>> >>
>>>>>> >> This is not really a problem on XenServer because XenServer
does
>>>>>> >> not
>>>>>> >>store VM config files in the SR, so I think we're OK there.
>>>>>> >>
>>>>>> >> We should also be OK for KVM.
>>>>>> >>
>>>>>> >> * Technically it can have many if those other VMDK files
are delta
>>>>>> >>snapshots, but they still - together - represent a single
disk.
>>>>>> >>
>>>>>> >>
>>>>>> >> On Fri, Mar 28, 2014 at 10:36 AM, Alena Prokharchyk
>>>>>> >><Alena.Prokharchyk@citrix.com<mailto:Alena.Prokharchyk@citrix.com>>
>>>>>> >>wrote:
>>>>>> >> Mike, thank you for the explanation on managed storage..
As far as
>>>>>> >> I
>>>>>> >>understand from your email, the main difference is instead
of
>>>>>> >> creating
>>>>>> >>an SR on the PS, CloudStack will recognize pre-existing volume
>>>>>> >> created
>>>>>> >>outside of the CS. Am I correct?
>>>>>> >>
>>>>>> >> If so, I don't think there would be any difference. When
root
>>>>>> >> volume
>>>>>> >>detach happens, no storage attributes - path, clusterId -
are being
>>>>>> >>changed. And we would apply the same set of checks to the
root
>>>>>> >> volume
>>>>>> >>attach, as for a dataDisk attach.
>>>>>> >>
>>>>>> >> -Alena.
>>>>>> >>
>>>>>> >> From: Mike Tutkowski
>>>>>> >><mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>>
>>>>>> >> Date: Thursday, March 27, 2014 at 9:40 PM
>>>>>> >> To: Alena Prokharchyk
>>>>>> >><alena.prokharchyk@citrix.com<mailto:alena.prokharchyk@citrix.com>>
>>>>>> >> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>>>>>> >><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>>>>>> >> Subject: Re: [PROPOSAL] ROOT volume detach - feature for
CS 4.5
>>>>>> >>
>>>>>> >> Hi Alena,
>>>>>> >>
>>>>>> >> I was wondering if you've taken "managed" storage into
>>>>>> >> consideration
>>>>>> >>for this?
>>>>>> >>
>>>>>> >> If you're unfamiliar with it, managed storage is named as
such
>>>>>> >> because
>>>>>> >>CloudStack manages it on behalf of the admin (ex. dynamically
>>>>>> >> creating
>>>>>> >>SRs as needed).
>>>>>> >>
>>>>>> >> For example, when I add primary storage to CloudStack that
is based
>>>>>> >> on
>>>>>> >>the SolidFire SAN, I use the SolidFire plug-in, which is
an example
>>>>>> >> of
>>>>>> >>managed storage.
>>>>>> >>
>>>>>> >> In this case, the primary storage represents a SAN as opposed
to a
>>>>>> >>preallocated volume.
>>>>>> >>
>>>>>> >> When the time comes to, say, attach a data disk to a VM
for the
>>>>>> >> first
>>>>>> >>time, the SolidFire plug-in goes off to its SAN and dynamically
>>>>>> >> creates
>>>>>> >>a new volume on it (with the appropriate size and IOPS
>>>>>> >> requirements).
>>>>>> >>
>>>>>> >> CloudStack has logic that recognizes managed storage.
>>>>>> >>
>>>>>> >> For example, for XenServer, its logic has been augmented
to
>>>>>> >>automatically create an SR based on the iSCSI target that
was
>>>>>> >> created on
>>>>>> >>the SAN and to create a VDI within it that is attached to
the VM in
>>>>>> >>question.
>>>>>> >>
>>>>>> >> The big takeaway is that each CloudStack volume here will
be
>>>>>> >> associated
>>>>>> >>with a unique volume on a SAN and consumed as an SR (XenServer)
or
>>>>>> >>datastore (ESX) (KVM handles this differently). In this situation,
>>>>>> >> there
>>>>>> >>is a 1:1 mapping between a SAN volume and an SR. No other
VDIs are
>>>>>> >>stored on the SR except for the one representing this one
CloudStack
>>>>>> >>volume.
>>>>>> >>
>>>>>> >> That being the case, I was wondering what you thought of
this with
>>>>>> >>regards to your root-volume-detach feature?
>>>>>> >>
>>>>>> >> If we don't want to look into this for 4.5, it might be
best to
>>>>>> >> simply
>>>>>> >>fail to detach a root volume from a VM if the volume is based
on
>>>>>> >> managed
>>>>>> >>storage or to fail to attach a bootable volume to a VM if
it is
>>>>>> >> based on
>>>>>> >>managed storage.
>>>>>> >>
>>>>>> >> Talk to you later,
>>>>>> >> Mike
>>>>>> >>
>>>>>> >>
>>>>>> >> On Tue, Mar 25, 2014 at 1:24 PM, Alena Prokharchyk
>>>>>> >><Alena.Prokharchyk@citrix.com<mailto:Alena.Prokharchyk@citrix.com>>
>>>>>> >>wrote:
>>>>>> >> Mike,
>>>>>> >>
>>>>>> >> Volume has a template_id referencing vm_template table.
Vm_template
>>>>>> >> has
>>>>>> >> bootable flag, so we will derive information from there.
>>>>>> >> And sure, this information will not change if the root disk
is
>>>>>> >> detached.
>>>>>> >>
>>>>>> >> On 3/25/14, 12:18 PM, "Mike Tutkowski"
>>>>>> >><mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>>
>>>>>> >> wrote:
>>>>>> >>
>>>>>> >>>Hi Alena,
>>>>>> >>>
>>>>>> >>>I was wondering how we plan to keep track of the new
"bootable"
>>>>>> >>>property?
>>>>>> >>>When we create a VM, would we just mark its root disk
as bootable
>>>>>> >>> and
>>>>>> >>>then
>>>>>> >>>that property becomes immutable (for the upgrade case,
all root
>>>>>> >>> disks
>>>>>> >>>would
>>>>>> >>>be marked as bootable)?
>>>>>> >>>
>>>>>> >>>I'm thinking we'd want to keep track of bootable disks
even when
>>>>>> >>> there
>>>>>> >>>are
>>>>>> >>>detached and turned into data disks. Is that what you
had in mind?
>>>>>> >>>
>>>>>> >>>Thanks!
>>>>>> >>>Mike
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>On Tue, Mar 25, 2014 at 12:20 PM, Alena Prokharchyk <
>>>>>> >>>Alena.Prokharchyk@citrix.com<mailto:Alena.Prokharchyk@citrix.com>>
>>>>>> >>>wrote:
>>>>>> >>>
>>>>>> >>>> Here is the link to the corresponding FS (placed
in "4.5 Design
>>>>>> >>>>documents"
>>>>>> >>>> section)
>>>>>> >>>>
>>>>>> >>>>
>>>>>>
>>>>>> >>>> >>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/ROOT+volume+deta
>>>>>> >>>>ch
>>>>>> >>>>
>>>>>> >>>> -Alena.
>>>>>> >>>>
>>>>>> >>>> From: Alena Prokharchyk
>>>>>>
>>>>>> >>>> >>>><alena.prokharchyk@citrix.com<mailto:alena.prokharchyk@citrix.com><mail
>>>>>> >>>>to:
>>>>>> >>>>
>>>>>> >>>> alena.prokharchyk@citrix.com<mailto:alena.prokharchyk@citrix.com>>>
>>>>>> >>>> Date: Monday, March 24, 2014 at 11:37 AM
>>>>>> >>>> To:
>>>>>>
>>>>>> >>>> >>>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org><mailto:dev
>>>>>> >>>>@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>"
<
>>>>>> >>>>
>>>>>>
>>>>>> >>>> >>>>dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org><mailto:dev@
>>>>>> >>>>cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>>
>>>>>> >>>> Subject: [PROPOSAL] ROOT volume detach - feature
for CS 4.5
>>>>>> >>>>
>>>>>> >>>> I would like to propose a new feature for CS 4.5
- "ROOT volume
>>>>>> >>>>detach"
>>>>>> >>>>-
>>>>>> >>>> that enables support for following use cases:
>>>>>> >>>>
>>>>>> >>>> 1) Replace current ROOT volume with the new one
for  existing vm.
>>>>>> >>>> 2) Case when ROOT volume of vm1 gets corrupted,
and you want to
>>>>>> >>>> attach
>>>>>> >>>>it
>>>>>> >>>> to vm2 to run the recovery utils on it. With current
CS
>>>>>> >>>> implemntation,
>>>>>> >>>>you
>>>>>> >>>> have to perform several steps - create snapshot
of vm1's volume,
>>>>>> >>>>create
>>>>>> >>>> volume from snapshot, attach volume to the vm2.
New
>>>>>> >>>> implementation
>>>>>> >>>>will
>>>>>> >>>> merge it all to one step.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> With the planned implementation, once the ROOT volume
is
>>>>>> >>>> detached, it
>>>>>> >>>>can
>>>>>> >>>> be attached to any existing vm (with respect to
>>>>>> >>>> Admin/Domain/Physical
>>>>>> >>>> resources limitations), either as a DataDisk or
a Root disk.
>>>>>> >>>>
>>>>>> >>>> Amazon EC2 already has this functionality in place,
so I think CS
>>>>>> >>>>would
>>>>>> >>>> only benefit from having it. Storage experts (Edison,
others)
>>>>>> >>>> please
>>>>>> >>>>raise
>>>>>> >>>> your concerns if you have any, or if you see any
potential
>>>>>> >>>> problems
>>>>>> >>>>with
>>>>>> >>>> the planned implementation. And if anyone can think
of other use
>>>>>> >>>> cases
>>>>>> >>>>this
>>>>>> >>>> feature can possible solve, I would appreciate this
input as
>>>>>> >>>> well.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> Feature limitations:
>>>>>> >>>>
>>>>>> >>>> * ROOT volume can be detached only when vm is in
Stopped state
>>>>>> >>>> * CS will fail to start the vm not having a ROOT
volume
>>>>>> >>>>
>>>>>> >>>> I will send out the link to the FS once I start
getting feedback
>>>>>> >>>> on
>>>>>> >>>>the
>>>>>> >>>> proposal.
>>>>>> >>>>
>>>>>> >>>> -Alena.
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>--
>>>>>> >>>*Mike Tutkowski*
>>>>>> >>>*Senior CloudStack Developer, SolidFire Inc.*
>>>>>> >>>e:
>>>>>> >>> mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>
>>>>>> >>>o: 303.746.7302<tel:303.746.7302>
>>>>>> >>>Advancing the way the world uses the
>>>>>> >>>cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>> >>>*(tm)*
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Mike Tutkowski
>>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>>> >> e:
>>>>>> >> mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>
>>>>>> >> o: 303.746.7302<tel:303.746.7302>
>>>>>> >> Advancing the way the world uses the
>>>>>> >>cloud<http://solidfire.com/solution/overview/?video=play>(tm)
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Mike Tutkowski
>>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>>> >> e:
>>>>>> >> mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>
>>>>>> >> o: 303.746.7302<tel:303.746.7302>
>>>>>> >> Advancing the way the world uses the
>>>>>> >>cloud<http://solidfire.com/solution/overview/?video=play>(tm)
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Mike Tutkowski
>>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>>> >> e:
>>>>>> >> mike.tutkowski@solidfire.com<mailto:mike.tutkowski@solidfire.com>
>>>>>> >> o: 303.746.7302
>>>>>> >> Advancing the way the world uses the
>>>>>> >>cloud<http://solidfire.com/solution/overview/?video=play>(tm)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Tutkowski
>>>>> Senior CloudStack Developer, SolidFire Inc.
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud(tm)
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mike Tutkowski
>>>> Senior CloudStack Developer, SolidFire Inc.
>>>> e: mike.tutkowski@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud(tm)
>>>
>>>
>>>
>>>
>>> --
>>> Mike Tutkowski
>>> Senior CloudStack Developer, SolidFire Inc.
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud(tm)
>>
>>
>>
>>
>> --
>> Mike Tutkowski
>> Senior CloudStack Developer, SolidFire Inc.
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud(tm)
>
>
>
>
> --
> Mike Tutkowski
> Senior CloudStack Developer, SolidFire Inc.
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud(tm)

Mime
View raw message