cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: add connect method on KVM storage code
Date Sat, 28 Sep 2013 01:31:12 GMT
Well, the default KVM one does, but only because the hypervisor is playing
the part or storage API. My plugin certainly doesn't have to talk to the
hypervisor, and I don't think I've instructed you to.
On Sep 27, 2013 7:29 PM, "Marcus Sorensen" <shadowsor@gmail.com> wrote:

> The plugin itself doesn't talk to the hypervisor.
> On Sep 27, 2013 7:28 PM, "Mike Tutkowski" <mike.tutkowski@solidfire.com>
> wrote:
>
>> Yeah, we should bring John Burwell into this conversation. He had a strong
>> opposition to storage plug-ins talking to hypervisors. Perhaps he was just
>> referring to shared storage, though (not local disks).
>>
>> Anyways, prior to 4.2 for XenServer and VMware, you had to preallocate
>> your
>> SR/datastore (there was a minor exception for XenServer). CloudStack
>> wasn't
>> involved in that process at all. It was a manual admin task.
>>
>> The storage plug-in framework didn't really solve that problem, though.
>> All
>> it did was allow you to create your storage volume and make it available
>> to
>> CloudStack. There was no way to get XenServer or VMware to actually use it
>> because they didn't want an IQN or whatever, they wanted an SR/datastore.
>>
>> That being the case, I had to write code to detect this "managed"
>> situation
>> in XenServer and VMware (in the attach/detach methods).
>>
>> For example, for XenServer I had to first create an SR and a VDI (which
>> the
>> SolidFire plug-in probably should have no knowledge of). Then the attach
>> logic would work.
>>
>> After a detach, I had to have XenServer "forget" the SR.
>>
>> What I was thinking is that the attach/detach methods aren't really the
>> right place for this special logic.
>>
>> We could have VolumeApiServiceImpl know about the special needs of
>> "managed" storage.
>>
>> If you want to do an attach of a volume and you're managed storage,
>> VolumeApiService Imple could first tell the hypervisor to create the
>> necessary objects (XenServer = SR/VDI, VMware = datastore/VMDK, KVM = just
>> establish the connection to the iSCSI target).
>>
>> It true that there are commands like CreateObjectCommand in XenServer and
>> VMware now, but they are only used when the storage is preallocated (not
>> when the storage is managed by CloudStack).
>>
>> In the end, we kind of have two different storage models in CloudStack:
>> Non-managed (preallocated storage by admin) and managed (plug-in
>> dynamically creates storage on a storage system).
>>
>>
>> On Fri, Sep 27, 2013 at 6:46 PM, Marcus Sorensen <shadowsor@gmail.com
>> >wrote:
>>
>> > On Fri, Sep 27, 2013 at 6:03 PM, Marcus Sorensen <shadowsor@gmail.com>
>> > wrote:
>> > > On Fri, Sep 27, 2013 at 5:16 PM, Mike Tutkowski
>> > > <mike.tutkowski@solidfire.com> wrote:
>> > >> createAsync is just for creating the SAN (or whatever storage)
>> volume.
>> > >> deleteAsync is the reverse.
>> > >
>> > > Exactly. It used to be that the hypervisor created the disk/lun/file
>> > > volume via createPhysicalDisk. Now it's done on the SAN if the plugin
>> > > supports it. Ideally, only calls that are required for utilizing the
>> > > storage (or perhaps things like copy to NFS, where a server need be
>> > > involved if your SAN can't do it directly) will go to the hypervisor,
>> > > for external plugins.
>> > >
>> > > So mgmt server creates the LUN on the SAN, then calls the hypervisor
>> > > to attach it to the host, so that a VM can make use of it. The
>> > > createAsync is hypervisor agnostic, it just creates a LUN, and then
>> > > when you go to start up a VM or attach it to one it calls the
>> > > hypervisor-specific code to make it available.
>> >
>> > I think this is the same with the other hypervisors as well. For
>> > example with Xen, you create the LUN via your plugin, and then a call
>> > is made to Xen to register it as an SR. It's basically the same, only
>> > you're now coding the 'register' part yourself (and it's ephemeral,
>> > it's re-registered whenever it's used). You basically take over some
>> > of the API coding that already exists on the other platforms, which
>> > means you can do whatever you want instead of just what's supported,
>> > but you have to do the work as well.
>> >
>> > >
>> > >>
>> > >> Technically even the default plug-in should not call into the
>> hypervisor
>> > >> layer.
>> > >
>> > > There's no way you can create a local storage file to use as a volume,
>> > > or CLVM volume, or other types of libvirt storage without calling a
>> > > service that runs on the hypervisor. Those things exist only on the
>> > > hypervisor, and are controlled by the hypervisor. For NFS, you could
>> > > create a separate API for your NFS server that creates qcow2 images on
>> > > your NFS primary, I suppose.
>> > >
>> > > One of the really nice things about KVM is that we can do whatever a
>> > > Linux box is capable of, it was one of the draws we had to it. We
>> > > wouldn't be able to do the storage we do with Xen or VMWare.
>> > >
>> > >>
>> > >> The storage layer should probably not be aware of the hypervisor
>> layer.
>> > >
>> > > That's fine, but there's no reason why a storage plugin can't talk to
>> > > the agent that happens to be running on the hypervisor for
>> > > implementation, if that's what the plugin intends.  I don't see the
>> > > distinction between utilizing the kvm agent as you storage API or
>> > > talking to a custom SAN API, or some other concocted service. That's
>> > > sort of the point of the plugin, people can do whatever they want.
>> > >
>> > >>
>> > >> On Fri, Sep 27, 2013 at 5:14 PM, Mike Tutkowski <
>> > >> mike.tutkowski@solidfire.com> wrote:
>> > >>
>> > >>> Well, from what I saw with XenServer and VMware, that hypervisor
>> > logic's
>> > >>> attachVolume command also assumed a VDI/VMDK was created in advance.
>> > >>>
>> > >>> I had to put logic in those attachVolume methods to create the
>> SR/VDI
>> > or
>> > >>> datastore/VMDK.
>> > >>>
>> > >>> However, thinking back on it, it might have made more sense for the
>> > >>> storage framework to detect if the storage in question was managed
>> and
>> > -
>> > >>> before calling attach - call create.
>> > >>>
>> > >>> If that logic was in place, I could have left attach/detachVolume
>> alone
>> > >>> and implemented create and delete in the hypervisor code to create
>> my
>> > >>> SR/VDI or datastore/VMDK.
>> > >>>
>> > >>> That makes sense to me because it is CloudStack-managed storage (so
>> > >>> CloudStack is calling into the hypervisor to create and delete these
>> > types
>> > >>> of objects...it's managing them).
>> > >>>
>> > >>>
>> > >>> On Fri, Sep 27, 2013 at 5:00 PM, Marcus Sorensen <
>> shadowsor@gmail.com
>> > >wrote:
>> > >>>
>> > >>>> On Fri, Sep 27, 2013 at 4:22 PM, Mike Tutkowski
>> > >>>> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> > Sure, sounds good - let me know when it's up on Review Board and
>> I
>> > can
>> > >>>> take
>> > >>>> > a look.
>> > >>>> >
>> > >>>> > I made most of the changes you and I talked about:
>> > >>>> >
>> > >>>> >
>> > >>>>
>> >
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/eb9b2edfc9062f9ca7961fecd5379b180ca3aed1
>> > >>>> >
>> > >>>> > I have a new idea, though, that I think will simplify this:
>> > >>>> >
>> > >>>> > The main "weirdness" right now is when attachVolume is called
>> that
>> > the
>> > >>>> > original logic assumed createVolume had been called already.
>> > >>>> >
>> > >>>> > In my case, this doesn't apply, so we had to place extra logic in
>> > >>>> > attachVolume to essentially "create" a volume. We decided to
>> make a
>> > >>>> connect
>> > >>>> > method, which establishes the iSCSI connection and creates a
>> > >>>> > KVMPhysicalDisk that can be returned when attachVolume calls
>> > >>>> > getPhysicalDisk.
>> > >>>> >
>> > >>>> > The "normal" place where you'd create a KVMPhysicalDisk, however,
>> > would
>> > >>>> be
>> > >>>> > in the createVolume method. Since I don't currently "create" a
>> > volume,
>> > >>>> my
>> > >>>> > only chance to note the size of the volume is in the connect
>> method.
>> > >>>>
>> > >>>> I don't think createVolume applies to plugins. My impression wash
>> that
>> > >>>> createAsync is called on the mgmt server side. If createVolume IS
>> > >>>> being called, that's weird. The idea here is that mgmt server
>> creates
>> > >>>> the LUN, and then on the KVM side attach is called (or
>> StartCommand if
>> > >>>> it's a root volume and vm is being started), and it assumes that
>> the
>> > >>>> LUN is already there, so we call connectPhysicalDisk to attach it
>> to
>> > >>>> the KVM host.
>> > >>>>
>> > >>>> >
>> > >>>> > It ends up being kind of weird to pass a size into the connect
>> > method,
>> > >>>> as
>> > >>>> > you've noted.
>> > >>>> >
>> > >>>> > What if we essentially left the attachVolume and detachVolume
>> > methods
>> > >>>> alone
>> > >>>> > (as in how they were before my changes)? We could have
>> > >>>> > VolumeApiServiceImpl, before sending the AttachCommand, detect if
>> > the
>> > >>>> > storage in question is managed. If it is, VolumeApiServiceImpl
>> could
>> > >>>> send a
>> > >>>> > CreateObjectCommand. I would then implement createPhysicalDisk to
>> > >>>> connect
>> > >>>> > my iSCSI target and create a KVMPhysicalDisk.
>> > >>>> >
>> > >>>> > On the reverse side, VolumeApiServiceImpl, after sending the
>> > >>>> DetachCommand,
>> > >>>> > could detect if the storage in question is managed. If it is,
>> > >>>> > VolumeApiServiceImpl could send a DeleteCommand. I would then
>> > implement
>> > >>>> the
>> > >>>> > deletePhysicalDisk method to disconnect my iSCSI session.
>> > >>>> >
>> > >>>> > What do you think?
>> > >>>>
>> > >>>> Maybe I'm just confused, but I thought the create and delete on the
>> > >>>> KVM side only apply to the default storage plugin, which has to
>> pass
>> > >>>> everything on the agent. I thought the creation/deletion of LUNs
>> > >>>> occured via createAsync and deleteAsync in your plugin.
>> > >>>>
>> > >>>> >
>> > >>>> >
>> > >>>> > On Fri, Sep 27, 2013 at 3:21 PM, Marcus Sorensen <
>> > shadowsor@gmail.com
>> > >>>> >wrote:
>> > >>>> >
>> > >>>> >> Ok, I've got our plugin working against 4.2. Tested start vm,
>> stop
>> > vm,
>> > >>>> >> migrate vm, attach volume, detach volume.  Other functions that
>> we
>> > >>>> >> already had in our StorageAdaptor implementation, such as
>> copying
>> > >>>> >> templates to primary storage, just worked without any
>> modification
>> > >>>> >> from our 4.1 version.
>> > >>>> >>
>> > >>>> >> I'll post a patch to reviewboard with the applicable changes. I
>> was
>> > >>>> >> correct that attachVolume and dettachVolume only apply to
>> > >>>> >> adding/removing disks from running VMs, so there were some more
>> > >>>> >> changes to LibvirtComputingResource. I don't intend for this
>> patch
>> > to
>> > >>>> >> be applied (for one it's against 4.2), but I want you to take a
>> > look
>> > >>>> >> and see if it will work for you as well. If it does, then it's a
>> > good
>> > >>>> >> indicator that it should work for other plugins too, or if it
>> > needs to
>> > >>>> >> be tweaked we can work it out.
>> > >>>> >>
>> > >>>> >> The gist is that we needed a connectPhysicalDisk call that can
>> > accept
>> > >>>> >> the pool/volume info (which we've discussed), but also a
>> version of
>> > >>>> >> connectPhysicalDisk that can take a vm specification
>> > >>>> >> (VirtualMachineTO) and figure out which pools/disks are needed
>> and
>> > >>>> >> attach them. I largely copied the code we had custom inserted
>> into
>> > our
>> > >>>> >> 4.1 and put it into KVMStoragePoolManager so that it will be
>> > adaptor
>> > >>>> >> agnostic.
>> > >>>> >>
>> > >>>> >> Same goes for disconnectPhysicalDisk.
>> > >>>> >>
>> > >>>> >> We also needed to pass the VirtualMachineTO in a few other
>> places
>> > like
>> > >>>> >> MigrateCommand and StopCommand, it's otherwise hard to know
>> which
>> > >>>> >> storage adaptors we need to deal with when all we have is a vm
>> > name or
>> > >>>> >> something like that.
>> > >>>> >>
>> > >>>> >> On Fri, Sep 27, 2013 at 12:56 AM, Mike Tutkowski
>> > >>>> >> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> > Thanks for the clarification on how that works.
>> > >>>> >> >
>> > >>>> >> > Also, yeah, I think CHAP only grants you access to a volume.
>> If
>> > >>>> multiple
>> > >>>> >> > hosts are using the CHAP credentials for a single volume, it's
>> > up to
>> > >>>> >> those
>> > >>>> >> > hosts to make sure they don't step on each other's toes (and
>> > this is
>> > >>>> - to
>> > >>>> >> > my understanding - how it works with XenServer and VMware).
>> > >>>> >> >
>> > >>>> >> >
>> > >>>> >> > On Fri, Sep 27, 2013 at 12:45 AM, Marcus Sorensen <
>> > >>>> shadowsor@gmail.com
>> > >>>> >> >wrote:
>> > >>>> >> >
>> > >>>> >> >> On Fri, Sep 27, 2013 at 12:21 AM, Mike Tutkowski
>> > >>>> >> >> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> > Maybe I should seek a little clarification as to how live
>> > >>>> migration
>> > >>>> >> works
>> > >>>> >> >> > in CS with KVM.
>> > >>>> >> >> >
>> > >>>> >> >> > Before we do a live migration of VM 1 from Host 1 to Host
>> 2,
>> > do we
>> > >>>> >> detach
>> > >>>> >> >> > all disks from VM1?
>> > >>>> >> >> >
>> > >>>> >> >> > If so, then we're good to go there.
>> > >>>> >> >> >
>> > >>>> >> >> > I'm not as clear with HA.
>> > >>>> >> >>
>> > >>>> >> >> During live migration this is what we currently do in our
>> > modified
>> > >>>> >> >> 4.1, I'm not sure if the new framework is set up for this,
>> but
>> > it
>> > >>>> >> >> should be made to do this if not.
>> > >>>> >> >>
>> > >>>> >> >> PrepareForMigrationCommand is called on destination host. In
>> > >>>> >> >> PrepareForMigrationCommand we added a few lines to call
>> > >>>> >> >> connectPhysicalDisk. This host connects the SAN disks to this
>> > new
>> > >>>> >> >> host, then creates a paused VM.
>> > >>>> >> >>
>> > >>>> >> >> MigrateCommand is called on the source host. This sends the
>> > proper
>> > >>>> >> >> command to transfer VM memory, then atomically cuts over to
>> the
>> > >>>> >> >> destination host. During this time, the disks are attached on
>> > both
>> > >>>> >> >> sides, but the VM is still the only thing that is using them,
>> > and it
>> > >>>> >> >> atomically cuts over. There's no caching on the host (qemu is
>> > using
>> > >>>> >> >> directio), so this is safe.
>> > >>>> >> >>
>> > >>>> >> >> After MigrateCommand completes it's VM passoff, we detach the
>> > disks
>> > >>>> >> >> before returning.
>> > >>>> >> >>
>> > >>>> >> >> >
>> > >>>> >> >> > If VM 1 goes down because Host 1 crashes, is the
>> attach-volume
>> > >>>> command
>> > >>>> >> >> > invoked as many times as need be (depending on how many
>> > volumes
>> > >>>> need
>> > >>>> >> to
>> > >>>> >> >> be
>> > >>>> >> >> > attached) when VM 1 is restarted on Host 2?
>> > >>>> >> >>
>> > >>>> >> >> From what I can tell, the attachVolume and dettachVolume
>> seemed
>> > to
>> > >>>> >> >> only be for attaching disks to existing, running VMs (i.e.
>> > inserting
>> > >>>> >> >> new XML into an existing domain definition).  Normally when
>> > >>>> starting a
>> > >>>> >> >> vm from scratch the vm definition, along with any currently
>> > attached
>> > >>>> >> >> disks, is passed in to StartCommand (which would also be
>> called
>> > >>>> during
>> > >>>> >> >> HA restart of a VM). In our 4.1 branch we also have a call to
>> > >>>> >> >> connectPhysicalDisk here, where we loop through the disk
>> > definitions
>> > >>>> >> >> that were passed.
>> > >>>> >> >>
>> > >>>> >> >> Again, I should be able to flesh out the differences in 4.2
>> and
>> > how
>> > >>>> to
>> > >>>> >> >> go about making this suitable for everyone in the coming
>> days,
>> > so
>> > >>>> long
>> > >>>> >> >> as you and anyone else writing plugins agree with the
>> changes.
>> > >>>> >> >>
>> > >>>> >> >> These processes would make sure the disks are available on
>> the
>> > hosts
>> > >>>> >> >> they need to be, but they don't really provide locking or
>> ensure
>> > >>>> that
>> > >>>> >> >> only the necessary hosts can write to or see the disks at any
>> > given
>> > >>>> >> >> time. I don't think CHAP does that either. We currently
>> generate
>> > >>>> ACLs
>> > >>>> >> >> via our SAN api during connectPhysicalDisk as a safety
>> measure,
>> > but
>> > >>>> if
>> > >>>> >> >> CloudStack is working properly it will be in charge of
>> > controlling
>> > >>>> >> >> that the disks are only being used where they should be. The
>> > ACLs
>> > >>>> just
>> > >>>> >> >> ensure that if the VM somehow gets started in two different
>> > places
>> > >>>> >> >> (e.g. HA malfunction), only one of them will have access to
>> the
>> > >>>> disks.
>> > >>>> >> >>
>> > >>>> >> >> >
>> > >>>> >> >> >
>> > >>>> >> >> > On Fri, Sep 27, 2013 at 12:17 AM, Mike Tutkowski <
>> > >>>> >> >> > mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >
>> > >>>> >> >> >> Let me clarify this line a bit:
>> > >>>> >> >> >>
>> > >>>> >> >> >> "We get away without this with XenServer and VMware
>> because
>> > - as
>> > >>>> far
>> > >>>> >> as
>> > >>>> >> >> I
>> > >>>> >> >> >> know - CS delegates HA and live migration to those
>> clusters
>> > and
>> > >>>> they
>> > >>>> >> >> handle
>> > >>>> >> >> >> it most likely with some kind of locking protocol on the
>> > >>>> >> SR/datastore."
>> > >>>> >> >> >>
>> > >>>> >> >> >> When I set up a XenServer or a VMware cluster, all nodes
>> in
>> > the
>> > >>>> >> cluster
>> > >>>> >> >> >> have the proper CHAP credentials and can access a shared
>> > >>>> >> SR/datastore.
>> > >>>> >> >> >>
>> > >>>> >> >> >> HA and live migrations are OK here because the cluster
>> > controls
>> > >>>> >> access
>> > >>>> >> >> to
>> > >>>> >> >> >> the VDI on the SR (or VMDK on the datastore) with some
>> kind
>> > of
>> > >>>> >> locking
>> > >>>> >> >> >> protocol, I expect.
>> > >>>> >> >> >>
>> > >>>> >> >> >> Since KVM isn't really in a cluster outside of the
>> CloudStack
>> > >>>> world,
>> > >>>> >> >> >> CloudStack has to handle these intricacies.
>> > >>>> >> >> >>
>> > >>>> >> >> >> In my case, I'm just presenting a raw disk to a VM on a
>> KVM
>> > host.
>> > >>>> >> >> >>
>> > >>>> >> >> >> In that case, HA and live migration depend on the storage
>> > plug-in
>> > >>>> >> being
>> > >>>> >> >> >> able to grant and revoke access to the volume for hosts as
>> > >>>> needed.
>> > >>>> >> >> >>
>> > >>>> >> >> >> I'd actually rather not even bother with CHAP when using
>> KVM.
>> > >>>> >> >> >>
>> > >>>> >> >> >>
>> > >>>> >> >> >> On Fri, Sep 27, 2013 at 12:06 AM, Mike Tutkowski <
>> > >>>> >> >> >> mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>
>> > >>>> >> >> >>> Hey Marcus,
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> I agree that CHAP does not fulfill the same role as
>> fencing.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> I think we're going to have trouble with HA and live
>> > migrations
>> > >>>> on
>> > >>>> >> KVM
>> > >>>> >> >> if
>> > >>>> >> >> >>> the storage plug-in doesn't have a way of knowing when a
>> > host
>> > >>>> wants
>> > >>>> >> to
>> > >>>> >> >> >>> access a volume and when we want to revoke access to that
>> > >>>> volume.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> We get away without this with XenServer and VMware
>> because
>> > - as
>> > >>>> far
>> > >>>> >> as
>> > >>>> >> >> I
>> > >>>> >> >> >>> know - CS delegates HA and live migration to those
>> clusters
>> > and
>> > >>>> they
>> > >>>> >> >> handle
>> > >>>> >> >> >>> it most likely with some kind of locking protocol on the
>> > >>>> >> SR/datastore.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> As far as the path field is concerned, I should be able
>> to
>> > >>>> populate
>> > >>>> >> it
>> > >>>> >> >> >>> with the IQN of the volume in question.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> One problem I do see, however, is in the getPhysicalDisk
>> > method.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> How are you envisioning I keep track of KVMPhysicalDisks
>> > that I
>> > >>>> >> create
>> > >>>> >> >> in
>> > >>>> >> >> >>> my connect method?
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> Initially I was thinking I'd just keep them in a map.
>> > Storage
>> > >>>> pool
>> > >>>> >> UUID
>> > >>>> >> >> >>> to KVMPhysicalDisks.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> The problem is, how do I reconstruct that map if the
>> agent
>> > is
>> > >>>> >> restarted
>> > >>>> >> >> >>> (say the host crashes or is restarted).
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> For storage pools, we get a message
>> > (ModifyStoragePoolCommand)
>> > >>>> from
>> > >>>> >> the
>> > >>>> >> >> >>> CS MS to tell us about all of the relevant storage pools.
>> > With
>> > >>>> this
>> > >>>> >> >> >>> message, I can reconstruct my cache of storage pools. No
>> > problem
>> > >>>> >> there.
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> But how will I know which volumes belong to a given
>> storage
>> > >>>> pool if
>> > >>>> >> I
>> > >>>> >> >> >>> have to rebuild that map? How will I even know which
>> volumes
>> > >>>> are in
>> > >>>> >> >> use at
>> > >>>> >> >> >>> all?
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> Thanks
>> > >>>> >> >> >>>
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> On Thu, Sep 26, 2013 at 11:37 PM, Marcus Sorensen <
>> > >>>> >> shadowsor@gmail.com
>> > >>>> >> >> >wrote:
>> > >>>> >> >> >>>
>> > >>>> >> >> >>>> On Thu, Sep 26, 2013 at 10:23 PM, Mike Tutkowski
>> > >>>> >> >> >>>> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>>> > My comments are inline:
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> > On Thu, Sep 26, 2013 at 9:10 PM, Marcus Sorensen <
>> > >>>> >> >> shadowsor@gmail.com
>> > >>>> >> >> >>>> >wrote:
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >> Ok, let me digest this a bit. I got the github
>> responses
>> > >>>> but I'd
>> > >>>> >> >> also
>> > >>>> >> >> >>>> >> like to keep it on-list as well.
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >>  My initial thoughts are:
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >> 1) I don't think disk format and size are necessary
>> > >>>> parameters
>> > >>>> >> for
>> > >>>> >> >> >>>> >> connectPhysicalDisk, as the format can be determined
>> by
>> > the
>> > >>>> >> >> adaptor,
>> > >>>> >> >> >>>> >> and the size is set during the createAsync call in
>> the
>> > >>>> plugin.
>> > >>>> >> We
>> > >>>> >> >> >>>> >> really just need the disk path and the pool.
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> > [Mike T.] I agree, format is not needed. The only
>> reason
>> > I
>> > >>>> have
>> > >>>> >> size
>> > >>>> >> >> >>>> passed
>> > >>>> >> >> >>>> > in is because I need to create a KVMPhysicalDisk at
>> the
>> > end
>> > >>>> of
>> > >>>> >> the
>> > >>>> >> >> >>>> connect
>> > >>>> >> >> >>>> > method. Since this KVMPhysicalDisk is (in the code on
>> > GitHub)
>> > >>>> >> being
>> > >>>> >> >> >>>> used to
>> > >>>> >> >> >>>> > create our XML to attach the disk, I figured we'd need
>> > that
>> > >>>> size.
>> > >>>> >> >> The
>> > >>>> >> >> >>>> > KVMPhysicalDisk I produce from my implementation of
>> > >>>> >> getPhysicalDisk
>> > >>>> >> >> is
>> > >>>> >> >> >>>> not
>> > >>>> >> >> >>>> > as good because I don't know the size of the disk at
>> this
>> > >>>> point
>> > >>>> >> (I
>> > >>>> >> >> >>>> don't
>> > >>>> >> >> >>>> > keep that information around). The reason I don't keep
>> > that
>> > >>>> info
>> > >>>> >> >> >>>> around is
>> > >>>> >> >> >>>> > because I don't have a good way to reproduce that
>> info if
>> > >>>> the KVM
>> > >>>> >> >> host
>> > >>>> >> >> >>>> is
>> > >>>> >> >> >>>> > rebooted. We get info about storage pools in the form
>> of
>> > a
>> > >>>> >> >> >>>> > ModifyStoragePoolCommand, but nothing about the
>> volumes
>> > >>>> inside of
>> > >>>> >> >> the
>> > >>>> >> >> >>>> > storage pool.
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> getPhysicalDisk is called in a bunch of places. I'd rely
>> > on the
>> > >>>> >> >> >>>> connectPhysicalDisk to only make the block device appear
>> > on the
>> > >>>> >> host,
>> > >>>> >> >> >>>> and then getPhysicalDisk to find that block device and
>> > fill out
>> > >>>> >> things
>> > >>>> >> >> >>>> like disk size and path (the real path to the local
>> block
>> > >>>> device)
>> > >>>> >> for
>> > >>>> >> >> >>>> passing and creating the disk XML. Trust me, unless
>> things
>> > have
>> > >>>> >> >> >>>> changed significantly you need to be able to identify a
>> > given
>> > >>>> >> device
>> > >>>> >> >> >>>> as a specific local disk by whatever you are setting the
>> > 'path'
>> > >>>> >> >> >>>> attribute to be.  getPhysicalDisk will be called on your
>> > >>>> storage
>> > >>>> >> pool
>> > >>>> >> >> >>>> with simply the path attribute, and via your adaptor
>> with
>> > the
>> > >>>> pool
>> > >>>> >> and
>> > >>>> >> >> >>>> path.
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> So you may set path as some combination of iqn and
>> > target/pool
>> > >>>> >> info,
>> > >>>> >> >> >>>> or if iqn is enough to identify a unique block device
>> (in
>> > >>>> >> >> >>>> /dev/disk/by-id maybe?) on a host then just use that.
>> Path
>> > just
>> > >>>> >> needs
>> > >>>> >> >> >>>> to be something, anything, to identify the disk on the
>> > host. In
>> > >>>> >> >> >>>> getPhysicalDisk, identify the local block device
>> matching
>> > the
>> > >>>> info,
>> > >>>> >> >> >>>> create a new KVMPhysicalDisk with the local path, size,
>> > etc,
>> > >>>> and
>> > >>>> >> >> >>>> return it.
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >> 2) I thought this access group thing you mention are
>> the
>> > >>>> >> >> grantAccess
>> > >>>> >> >> >>>> >> and revokeAccess calls in the storage plugin 2.0
>> design
>> > >>>> doc. Was
>> > >>>> >> >> that
>> > >>>> >> >> >>>> >> not implemented?
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> > [Mike T.] Yeah, as I mentioned in an e-mail way back,
>> > those
>> > >>>> >> methods
>> > >>>> >> >> >>>> were
>> > >>>> >> >> >>>> > never implemented in 4.2. I think you said you were
>> > going to
>> > >>>> get
>> > >>>> >> >> around
>> > >>>> >> >> >>>> > them not being implemented by keeping certain logic
>> that
>> > >>>> talks to
>> > >>>> >> >> the
>> > >>>> >> >> >>>> SAN
>> > >>>> >> >> >>>> > in the agent. I don't think we want any
>> > SolidFire-specific
>> > >>>> code
>> > >>>> >> in
>> > >>>> >> >> the
>> > >>>> >> >> >>>> > agent, however, so I can't go that route. If those
>> > methods
>> > >>>> do not
>> > >>>> >> >> get
>> > >>>> >> >> >>>> > implemented in 4.3, then I will need to use CHAP
>> > credentials
>> > >>>> for
>> > >>>> >> KVM
>> > >>>> >> >> >>>> (just
>> > >>>> >> >> >>>> > like I did with XenServer and VMware).
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> I initially figured your StorageAdaptor implementation
>> > would
>> > >>>> be all
>> > >>>> >> >> >>>> solidfire specific, just like the mgmt server side
>> plugin
>> > is.
>> > >>>> If it
>> > >>>> >> >> >>>> can be generic to all iscsi storage then that's great. I
>> > agree
>> > >>>> that
>> > >>>> >> >> >>>> ideally the agent wouldn't be making API calls to your
>> > SAN. I
>> > >>>> don't
>> > >>>> >> >> >>>> think it should be necessary given that you're not
>> going to
>> > >>>> use the
>> > >>>> >> >> >>>> ACL route. I'm not sure CHAP fills the same purpose of
>> > fencing.
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >> I see you've  added getters/setters for the attach
>> cmd
>> > to
>> > >>>> pass
>> > >>>> >> the
>> > >>>> >> >> >>>> >> iscsi info you need. Would it perhaps be possible to
>> > send a
>> > >>>> >> details
>> > >>>> >> >> >>>> >> Map<String, String> instead? Then any plugin
>> implementer
>> > >>>> could
>> > >>>> >> >> attach
>> > >>>> >> >> >>>> >> arbitrary data they need. So it might be
>> > >>>> >> >> >>>> >> connectPhysicalDisk(StoragePoolType type, String
>> > poolUuid,
>> > >>>> >> String
>> > >>>> >> >> >>>> >> volPath, Map<String, String> details)?  I'll have to
>> > look
>> > >>>> and
>> > >>>> >> see
>> > >>>> >> >> >>>> >> where those cmd. attributes are set, ideally it
>> would be
>> > >>>> all the
>> > >>>> >> >> way
>> > >>>> >> >> >>>> >> back in the plugin to avoid custom code for every
>> > adaptor
>> > >>>> that
>> > >>>> >> >> wants
>> > >>>> >> >> >>>> >> to set details.
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> > [Mike T.] If I'm not using the volumes.path field for
>> > >>>> anything, I
>> > >>>> >> >> can
>> > >>>> >> >> >>>> stick
>> > >>>> >> >> >>>> > the IQN in volumes.path (as well as leaving it in
>> > >>>> >> >> volumes.iscsi_name,
>> > >>>> >> >> >>>> which
>> > >>>> >> >> >>>> > is used elsewhere). That way we only have to ask for
>> > >>>> getPath().
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> Yeah, whatever it is that you'd need to find the right
>> > block
>> > >>>> device
>> > >>>> >> >> >>>> should go in the path. If you look through
>> > >>>> LibvirtComputingResource
>> > >>>> >> >> >>>> you'll see stuff like this sprinkled around:
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>>                 KVMPhysicalDisk volume =
>> > >>>> >> >> primaryPool.getPhysicalDisk(cmd
>> > >>>> >> >> >>>>                         .getVolumePath());
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> or:
>> > >>>> >> >> >>>>         String volid = cmd.getPath();
>> > >>>> >> >> >>>>          KVMPhysicalDisk vol =
>> pool.getPhysicalDisk(volid);
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> or:
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>>                     KVMPhysicalDisk physicalDisk =
>> > >>>> >> >> >>>> _storagePoolMgr.getPhysicalDisk( store.getPoolType(),
>> > >>>> >> >> >>>>                             store.getUuid(),
>> > >>>> >> >> >>>>                             data.getPath());
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> Maybe some of it is short-circuited by the new
>> > >>>> KVMStorageProcessor,
>> > >>>> >> >> >>>> but I'd still implement a working one, and then
>> > attachVolume
>> > >>>> can
>> > >>>> >> call
>> > >>>> >> >> >>>> getPhysicalDisk after connectPhysicalDisk, even on your
>> > >>>> >> adaptor/pool.
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >> On Thu, Sep 26, 2013 at 7:35 PM, Mike Tutkowski
>> > >>>> >> >> >>>> >> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>>> >> > Also, if we went the non-CHAP route, before
>> attaching
>> > a
>> > >>>> volume
>> > >>>> >> >> to a
>> > >>>> >> >> >>>> VM,
>> > >>>> >> >> >>>> >> > we'd have to tell the plug-in to set up a volume
>> > access
>> > >>>> group.
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> > When a volume is detached from a VM, we'd have to
>> > tell the
>> > >>>> >> >> plug-in
>> > >>>> >> >> >>>> to
>> > >>>> >> >> >>>> >> > delete the volume access group.
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> > On Thu, Sep 26, 2013 at 7:32 PM, Mike Tutkowski <
>> > >>>> >> >> >>>> >> > mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> >> I mention this is my comments on GitHub, as well,
>> but
>> > >>>> CHAP
>> > >>>> >> info
>> > >>>> >> >> is
>> > >>>> >> >> >>>> >> >> associated with an account - not a storage pool.
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >> Ideally we could do without CHAP info entirely if
>> we
>> > had
>> > >>>> a
>> > >>>> >> >> >>>> reliable way
>> > >>>> >> >> >>>> >> to
>> > >>>> >> >> >>>> >> >> tell the storage plug-in that a given host wants
>> to
>> > >>>> access a
>> > >>>> >> >> given
>> > >>>> >> >> >>>> >> volume.
>> > >>>> >> >> >>>> >> >> In this case, my storage plug-in could create
>> what we
>> > >>>> call a
>> > >>>> >> >> Volume
>> > >>>> >> >> >>>> >> Access
>> > >>>> >> >> >>>> >> >> Group on the SAN. It would essentially say, "The
>> host
>> > >>>> with
>> > >>>> >> IQN
>> > >>>> >> >> <x>
>> > >>>> >> >> >>>> can
>> > >>>> >> >> >>>> >> >> access the volume with IQN <y> without using CHAP
>> > >>>> >> credentials."
>> > >>>> >> >> Of
>> > >>>> >> >> >>>> >> course
>> > >>>> >> >> >>>> >> >> we'd need a way to revoke this privilege in the
>> > event of
>> > >>>> a
>> > >>>> >> live
>> > >>>> >> >> >>>> >> migration
>> > >>>> >> >> >>>> >> >> of a VM. Right now, I do not believe such a
>> facility
>> > is
>> > >>>> >> >> supported
>> > >>>> >> >> >>>> with
>> > >>>> >> >> >>>> >> the
>> > >>>> >> >> >>>> >> >> storage plug-ins.
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >> On Thu, Sep 26, 2013 at 4:56 PM, Marcus Sorensen <
>> > >>>> >> >> >>>> shadowsor@gmail.com
>> > >>>> >> >> >>>> >> >wrote:
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >>> Looking at your code, is the chap info stored
>> with
>> > the
>> > >>>> pool,
>> > >>>> >> >> so we
>> > >>>> >> >> >>>> >> >>> could pass the pool to the adaptor? That would be
>> > more
>> > >>>> >> >> agnostic,
>> > >>>> >> >> >>>> >> >>> anyone implementing a plugin could pull the
>> > specifics
>> > >>>> they
>> > >>>> >> need
>> > >>>> >> >> >>>> for
>> > >>>> >> >> >>>> >> >>> their stuff out of the pool on the adaptor side,
>> > rather
>> > >>>> than
>> > >>>> >> >> >>>> creating
>> > >>>> >> >> >>>> >> >>> custom signatures.
>> > >>>> >> >> >>>> >> >>>
>> > >>>> >> >> >>>> >> >>> Also, I think we may want to consider
>> implementing
>> > >>>> >> >> >>>> connect/disconnect
>> > >>>> >> >> >>>> >> >>> as just dummy methods in LibvirtStorageAdaptor,
>> so
>> > we
>> > >>>> don't
>> > >>>> >> >> have
>> > >>>> >> >> >>>> to be
>> > >>>> >> >> >>>> >> >>> picky about which adaptors/types in every single
>> > place
>> > >>>> we
>> > >>>> >> may
>> > >>>> >> >> >>>> want to
>> > >>>> >> >> >>>> >> >>> connect/disconnect (in 4.1 there were several,
>> I'm
>> > not
>> > >>>> sure
>> > >>>> >> if
>> > >>>> >> >> >>>> >> >>> everything goes through this in 4.2). We can just
>> > call
>> > >>>> >> >> >>>> >> >>> adaptor.connectPhysicalDisk and the adaptor can
>> > decide
>> > >>>> if it
>> > >>>> >> >> >>>> needs to
>> > >>>> >> >> >>>> >> >>> do anything.
>> > >>>> >> >> >>>> >> >>>
>> > >>>> >> >> >>>> >> >>> Comments are attached to your commit, I just
>> wanted
>> > to
>> > >>>> echo
>> > >>>> >> >> them
>> > >>>> >> >> >>>> here
>> > >>>> >> >> >>>> >> >>> on-list.
>> > >>>> >> >> >>>> >> >>>
>> > >>>> >> >> >>>> >> >>> On Thu, Sep 26, 2013 at 4:32 PM, Mike Tutkowski
>> > >>>> >> >> >>>> >> >>> <mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>>> >> >>> > Oh, SnapshotTestWithFakeData is just modified
>> > because
>> > >>>> the
>> > >>>> >> >> code
>> > >>>> >> >> >>>> wasn't
>> > >>>> >> >> >>>> >> >>> > building until I corrected this. It has nothing
>> > >>>> really to
>> > >>>> >> do
>> > >>>> >> >> >>>> with my
>> > >>>> >> >> >>>> >> >>> real
>> > >>>> >> >> >>>> >> >>> > changes.
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> > On Thu, Sep 26, 2013 at 4:31 PM, Mike
>> Tutkowski <
>> > >>>> >> >> >>>> >> >>> > mike.tutkowski@solidfire.com> wrote:
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> >> Hey Marcus,
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >> I implemented your recommendations regarding
>> > adding
>> > >>>> >> connect
>> > >>>> >> >> and
>> > >>>> >> >> >>>> >> >>> disconnect
>> > >>>> >> >> >>>> >> >>> >> methods. It is not yet checked in (as you
>> know,
>> > >>>> having
>> > >>>> >> >> trouble
>> > >>>> >> >> >>>> with
>> > >>>> >> >> >>>> >> my
>> > >>>> >> >> >>>> >> >>> KVM
>> > >>>> >> >> >>>> >> >>> >> environment), but it is on GitHub here:
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>>
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>>
>> > >>>> >> >>
>> > >>>> >>
>> > >>>>
>> >
>> https://github.com/mike-tutkowski/incubator-cloudstack/commit/f2897c65689012e6157c0a0c2ed7e5355900c59a
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >> Please let me know if you have any more
>> comments.
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >> Thanks!
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >> On Thu, Sep 26, 2013 at 4:05 PM, Marcus
>> Sorensen
>> > <
>> > >>>> >> >> >>>> >> shadowsor@gmail.com
>> > >>>> >> >> >>>> >> >>> >wrote:
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>> Mike, everyone,
>> > >>>> >> >> >>>> >> >>> >>>    As I've mentioned on the board, I'm
>> working
>> > on
>> > >>>> >> getting
>> > >>>> >> >> our
>> > >>>> >> >> >>>> own
>> > >>>> >> >> >>>> >> >>> >>> internal KVM storage plugin working on 4.2.
>> In
>> > the
>> > >>>> >> >> interest of
>> > >>>> >> >> >>>> >> making
>> > >>>> >> >> >>>> >> >>> >>> it forward compatible, I just wanted to
>> confirm
>> > >>>> what you
>> > >>>> >> >> were
>> > >>>> >> >> >>>> doing
>> > >>>> >> >> >>>> >> >>> >>> with the solidfire plugin as far as attaching
>> > your
>> > >>>> iscsi
>> > >>>> >> >> >>>> LUNs. We
>> > >>>> >> >> >>>> >> had
>> > >>>> >> >> >>>> >> >>> >>> discussed a new connectPhysicalDisk method
>> for
>> > the
>> > >>>> >> >> >>>> StorageAdaptor
>> > >>>> >> >> >>>> >> >>> >>> class, something perhaps like:
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>> public boolean connectPhysicalDisk(String
>> > >>>> volumeUuid,
>> > >>>> >> >> >>>> >> KVMStoragePool
>> > >>>> >> >> >>>> >> >>> >>> pool);
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>> then added to KVMStoragePoolManager:
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>> public boolean
>> > connectPhysicalDisk(StoragePoolType
>> > >>>> type,
>> > >>>> >> >> >>>> String
>> > >>>> >> >> >>>> >> >>> >>> poolUuid, String volPath) {
>> > >>>> >> >> >>>> >> >>> >>>         StorageAdaptor adaptor =
>> > >>>> >> getStorageAdaptor(type);
>> > >>>> >> >> >>>> >> >>> >>>         KVMStoragePool pool =
>> > >>>> >> >> >>>> adaptor.getStoragePool(poolUuid);
>> > >>>> >> >> >>>> >> >>> >>>         return
>> > adaptor.connectPhysicalDisk(volPath,
>> > >>>> >> pool);
>> > >>>> >> >> >>>> >> >>> >>>     }
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>> Something similar to this for disconnect as
>> > well.
>> > >>>> Then
>> > >>>> >> in
>> > >>>> >> >> the
>> > >>>> >> >> >>>> >> >>> >>> KVMStorageProcessor these can be called as
>> > needed
>> > >>>> for
>> > >>>> >> >> >>>> >> attach/detach.
>> > >>>> >> >> >>>> >> >>> >>> We can probably stub out one in
>> > >>>> LibvirtStorageAdaptor so
>> > >>>> >> >> >>>> there's no
>> > >>>> >> >> >>>> >> >>> >>> need to switch or if/else for pool types, for
>> > >>>> example in
>> > >>>> >> >> >>>> >> >>> >>> KVMStorageProcessor.attachVolume.
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>> I have debated on whether or not it should
>> just
>> > be
>> > >>>> >> rolled
>> > >>>> >> >> into
>> > >>>> >> >> >>>> >> >>> >>> getPhysicalDisk, having it connect the disk
>> if
>> > it's
>> > >>>> not
>> > >>>> >> >> >>>> already
>> > >>>> >> >> >>>> >> >>> >>> connected. getPhysicalDisk is called a lot,
>> and
>> > I'm
>> > >>>> not
>> > >>>> >> >> sure
>> > >>>> >> >> >>>> it
>> > >>>> >> >> >>>> >> always
>> > >>>> >> >> >>>> >> >>> >>> needs to connect the disk when it does. In
>> past
>> > >>>> >> iterations
>> > >>>> >> >> >>>> >> >>> >>> getPhysicalDisk has simply spoken to our SAN
>> > api and
>> > >>>> >> >> returned
>> > >>>> >> >> >>>> the
>> > >>>> >> >> >>>> >> disk
>> > >>>> >> >> >>>> >> >>> >>> details, nothing more. So it seemed more
>> > flexible
>> > >>>> and
>> > >>>> >> >> >>>> granular to
>> > >>>> >> >> >>>> >> do
>> > >>>> >> >> >>>> >> >>> >>> the connectPhysicalDisk (we have one now in
>> our
>> > 4.1
>> > >>>> >> >> version).
>> > >>>> >> >> >>>> >> >>> >>>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >> --
>> > >>>> >> >> >>>> >> >>> >> *Mike Tutkowski*
>> > >>>> >> >> >>>> >> >>> >> *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>>> >> >>> >> e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>>> >> >>> >> o: 303.746.7302
>> > >>>> >> >> >>>> >> >>> >> Advancing the way the world uses the cloud<
>> > >>>> >> >> >>>> >> >>>
>> http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >>>> >> >>> >> *™*
>> > >>>> >> >> >>>> >> >>> >>
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> >
>> > >>>> >> >> >>>> >> >>> > --
>> > >>>> >> >> >>>> >> >>> > *Mike Tutkowski*
>> > >>>> >> >> >>>> >> >>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>>> >> >>> > e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>>> >> >>> > o: 303.746.7302
>> > >>>> >> >> >>>> >> >>> > Advancing the way the world uses the
>> > >>>> >> >> >>>> >> >>> > cloud<
>> > >>>> http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >>>> >> >>> > *™*
>> > >>>> >> >> >>>> >> >>>
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >> --
>> > >>>> >> >> >>>> >> >> *Mike Tutkowski*
>> > >>>> >> >> >>>> >> >> *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>>> >> >> e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>>> >> >> o: 303.746.7302
>> > >>>> >> >> >>>> >> >> Advancing the way the world uses the cloud<
>> > >>>> >> >> >>>> >> http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >>>> >> >> *™*
>> > >>>> >> >> >>>> >> >>
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> >
>> > >>>> >> >> >>>> >> > --
>> > >>>> >> >> >>>> >> > *Mike Tutkowski*
>> > >>>> >> >> >>>> >> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>>> >> > e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>>> >> > o: 303.746.7302
>> > >>>> >> >> >>>> >> > Advancing the way the world uses the
>> > >>>> >> >> >>>> >> > cloud<
>> > http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >>>> >> > *™*
>> > >>>> >> >> >>>> >>
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> >
>> > >>>> >> >> >>>> > --
>> > >>>> >> >> >>>> > *Mike Tutkowski*
>> > >>>> >> >> >>>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>>> > e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>>> > o: 303.746.7302
>> > >>>> >> >> >>>> > Advancing the way the world uses the
>> > >>>> >> >> >>>> > cloud<
>> http://solidfire.com/solution/overview/?video=play
>> > >
>> > >>>> >> >> >>>> > *™*
>> > >>>> >> >> >>>>
>> > >>>> >> >> >>>
>> > >>>> >> >> >>>
>> > >>>> >> >> >>>
>> > >>>> >> >> >>> --
>> > >>>> >> >> >>> *Mike Tutkowski*
>> > >>>> >> >> >>> *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >>> e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >>> o: 303.746.7302
>> > >>>> >> >> >>> Advancing the way the world uses the cloud<
>> > >>>> >> >> http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >>> *™*
>> > >>>> >> >> >>>
>> > >>>> >> >> >>
>> > >>>> >> >> >>
>> > >>>> >> >> >>
>> > >>>> >> >> >> --
>> > >>>> >> >> >> *Mike Tutkowski*
>> > >>>> >> >> >> *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> >> e: mike.tutkowski@solidfire.com
>> > >>>> >> >> >> o: 303.746.7302
>> > >>>> >> >> >> Advancing the way the world uses the cloud<
>> > >>>> >> >> http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> >> *™*
>> > >>>> >> >> >>
>> > >>>> >> >> >
>> > >>>> >> >> >
>> > >>>> >> >> >
>> > >>>> >> >> > --
>> > >>>> >> >> > *Mike Tutkowski*
>> > >>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> >> > e: mike.tutkowski@solidfire.com
>> > >>>> >> >> > o: 303.746.7302
>> > >>>> >> >> > Advancing the way the world uses the
>> > >>>> >> >> > cloud<http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> >> > *™*
>> > >>>> >> >>
>> > >>>> >> >
>> > >>>> >> >
>> > >>>> >> >
>> > >>>> >> > --
>> > >>>> >> > *Mike Tutkowski*
>> > >>>> >> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> >> > e: mike.tutkowski@solidfire.com
>> > >>>> >> > o: 303.746.7302
>> > >>>> >> > Advancing the way the world uses the
>> > >>>> >> > cloud<http://solidfire.com/solution/overview/?video=play>
>> > >>>> >> > *™*
>> > >>>> >>
>> > >>>> >
>> > >>>> >
>> > >>>> >
>> > >>>> > --
>> > >>>> > *Mike Tutkowski*
>> > >>>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > >>>> > e: mike.tutkowski@solidfire.com
>> > >>>> > o: 303.746.7302
>> > >>>> > Advancing the way the world uses the
>> > >>>> > cloud<http://solidfire.com/solution/overview/?video=play>
>> > >>>> > *™*
>> > >>>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> *Mike Tutkowski*
>> > >>> *Senior CloudStack Developer, SolidFire Inc.*
>> > >>> e: mike.tutkowski@solidfire.com
>> > >>> o: 303.746.7302
>> > >>> Advancing the way the world uses the cloud<
>> > http://solidfire.com/solution/overview/?video=play>
>> > >>> *™*
>> > >>>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> *Mike Tutkowski*
>> > >> *Senior CloudStack Developer, SolidFire Inc.*
>> > >> e: mike.tutkowski@solidfire.com
>> > >> o: 303.746.7302
>> > >> Advancing the way the world uses the
>> > >> cloud<http://solidfire.com/solution/overview/?video=play>
>> > >> *™*
>> >
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *™*
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message