Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CAC51008E for ; Wed, 18 Sep 2013 02:01:39 +0000 (UTC) Received: (qmail 15751 invoked by uid 500); 18 Sep 2013 02:01:38 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 15719 invoked by uid 500); 18 Sep 2013 02:01:38 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 15711 invoked by uid 99); 18 Sep 2013 02:01:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 02:01:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.214.174 as permitted sender) Received: from [209.85.214.174] (HELO mail-ob0-f174.google.com) (209.85.214.174) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2013 02:01:28 +0000 Received: by mail-ob0-f174.google.com with SMTP id uz6so6555531obc.19 for ; Tue, 17 Sep 2013 19:01:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=i77GAX1bE0GDQTyB4aAapAOCpIsKClZZg2QVoWrzVcU=; b=BLUzmMveo+vPGKgMu4yUKo+t9HiZLlXtixjKkjDyQto/CH6Sk5JUFKNtRxMqxClo53 Y2uND/45Z+gQYHScu3cCXwnPgLAf6KAwp+xv+V94WQwnEED18gpGzTalAz35H4UhLHXH /U2Lxp/0srIya5WaYFzzt6Mfua/uU4gcaDXiZNNV4bgBba59998SEzmfRtYNugQdP811 LSsrtcdr94OxmqVdpTZd39Nwzcg6tAv9J3wv5/br35kaYNFDAgxM0tk/2AcUtgy3WUYf q9d5Zhu5frdvYcco7SCTEkCJcDDkxeUM/Hger3XZ7dgonkNz0tPOj95bccVOIZ1tjTBq Cp+Q== X-Gm-Message-State: ALoCoQmmgVwtbKeUt+H2GvLQr/sF9HcVhFairQc0aHatq9WAAaiNyufY5s42k2b+ilAbI0FYrvF/ MIME-Version: 1.0 X-Received: by 10.182.129.201 with SMTP id ny9mr32711004obb.0.1379469666101; Tue, 17 Sep 2013 19:01:06 -0700 (PDT) Received: by 10.182.139.100 with HTTP; Tue, 17 Sep 2013 19:01:05 -0700 (PDT) In-Reply-To: References: Date: Tue, 17 Sep 2013 20:01:05 -0600 Message-ID: Subject: Re: Managed storage with KVM From: Mike Tutkowski To: "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=089e015366b46d8acc04e69ecfee X-Virus-Checked: Checked by ClamAV on apache.org --089e015366b46d8acc04e69ecfee Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Plus, when you log in to a LUN, you need the CHAP info and this info is required for each LUN (as opposed to being for the SAN). This is how my createStoragePool currently looks, so I think we're on the same page. public KVMStoragePool createStoragePool(String name, String host, int port, String path, String userInfo, StoragePoolType type) { iScsiAdmStoragePool storagePool =3D new iScsiAdmStoragePool(name, host, port, this); _mapUuidToAdaptor.put(name, storagePool); return storagePool; } On Tue, Sep 17, 2013 at 7:34 PM, Marcus Sorensen wrote= : > What do you do with Xen? I imagine the user enter the SAN details when > registering the pool? A the pool details are basically just instructions = on > how to log into a target, correct? > > You can choose to log in a KVM host to the target during createStoragePoo= l > and save the pool in a map, or just save the pool info in a map for futur= e > reference by uuid, for when you do need to log in. The createStoragePool > then just becomes a way to save the pool info to the agent. Personally, I= 'd > log in on the pool create and look/scan for specific luns when they're > needed, but I haven't thought it through thoroughly. I just say that main= ly > because login only happens once, the first time the pool is used, and eve= ry > other storage command is about discovering new luns or maybe > deleting/disconnecting luns no longer needed. On the other hand, you coul= d > do all of the above: log in on pool create, then also check if you're > logged in on other commands and log in if you've lost connection. > > With Xen, what does your registered pool show in the UI for avail/used > capacity, and how does it get that info? I assume there is some sort of > disk pool that the luns are carved from, and that your plugin is called t= o > talk to the SAN and expose to the user how much of that pool has been > allocated. Knowing how you already solves these problems with Xen will he= lp > figure out what to do with KVM. > > If this is the case, I think the plugin can continue to handle it rather > than getting details from the agent. I'm not sure if that means nulls are > OK for these on the agent side or what, I need to look at the storage > plugin arch more closely. > On Sep 17, 2013 7:08 PM, "Mike Tutkowski" > wrote: > > > Hey Marcus, > > > > I'm reviewing your e-mails as I implement the necessary methods in new > > classes. > > > > "So, referencing StorageAdaptor.java, createStoragePool accepts all of > > the pool data (host, port, name, path) which would be used to log the > > host into the initiator." > > > > Can you tell me, in my case, since a storage pool (primary storage) is > > actually the SAN, I wouldn't really be logging into anything at this > point, > > correct? > > > > Also, what kind of capacity, available, and used bytes make sense to > report > > for KVMStoragePool (since KVMStoragePool represents the SAN in my case > and > > not an individual LUN)? > > > > Thanks! > > > > > > On Fri, Sep 13, 2013 at 11:42 PM, Marcus Sorensen > >wrote: > > > > > Ok, KVM will be close to that, of course, because only the hypervisor > > > classes differ, the rest is all mgmt server. Creating a volume is jus= t > > > a db entry until it's deployed for the first time. AttachVolumeComman= d > > > on the agent side (LibvirtStorageAdaptor.java is analogous to > > > CitrixResourceBase.java) will do the iscsiadm commands (via a KVM > > > StorageAdaptor) to log in the host to the target and then you have a > > > block device. Maybe libvirt will do that for you, but my quick read > > > made it sound like the iscsi libvirt pool type is actually a pool, no= t > > > a lun or volume, so you'll need to figure out if that works or if > > > you'll have to use iscsiadm commands. > > > > > > If you're NOT going to use LibvirtStorageAdaptor (because Libvirt > > > doesn't really manage your pool the way you want), you're going to > > > have to create a version of KVMStoragePool class and a StorageAdaptor > > > class (see LibvirtStoragePool.java and LibvirtStorageAdaptor.java), > > > implementing all of the methods, then in KVMStorageManager.java > > > there's a "_storageMapper" map. This is used to select the correct > > > adaptor, you can see in this file that every call first pulls the > > > correct adaptor out of this map via getStorageAdaptor. So you can see > > > a comment in this file that says "add other storage adaptors here", > > > where it puts to this map, this is where you'd register your adaptor. > > > > > > So, referencing StorageAdaptor.java, createStoragePool accepts all of > > > the pool data (host, port, name, path) which would be used to log the > > > host into the initiator. I *believe* the method getPhysicalDisk will > > > need to do the work of attaching the lun. AttachVolumeCommand calls > > > this and then creates the XML diskdef and attaches it to the VM. Now, > > > one thing you need to know is that createStoragePool is called often, > > > sometimes just to make sure the pool is there. You may want to create > > > a map in your adaptor class and keep track of pools that have been > > > created, LibvirtStorageAdaptor doesn't have to do this because it ask= s > > > libvirt about which storage pools exist. There are also calls to > > > refresh the pool stats, and all of the other calls can be seen in the > > > StorageAdaptor as well. There's a createPhysical disk, clone, etc, bu= t > > > it's probably a hold-over from 4.1, as I have the vague idea that > > > volumes are created on the mgmt server via the plugin now, so whateve= r > > > doesn't apply can just be stubbed out (or optionally > > > extended/reimplemented here, if you don't mind the hosts talking to > > > the san api). > > > > > > There is a difference between attaching new volumes and launching a V= M > > > with existing volumes. In the latter case, the VM definition that wa= s > > > passed to the KVM agent includes the disks, (StartCommand). > > > > > > I'd be interested in how your pool is defined for Xen, I imagine it > > > would need to be kept the same. Is it just a definition to the SAN > > > (ip address or some such, port number) and perhaps a volume pool name= ? > > > > > > > If there is a way for me to update the ACL list on the SAN to have > > only a > > > > single KVM host have access to the volume, that would be ideal. > > > > > > That depends on your SAN API. I was under the impression that the > > > storage plugin framework allowed for acls, or for you to do whatever > > > you want for create/attach/delete/snapshot, etc. You'd just call your > > > SAN API with the host info for the ACLs prior to when the disk is > > > attached (or the VM is started). I'd have to look more at the > > > framework to know the details, in 4.1 I would do this in > > > getPhysicalDisk just prior to connecting up the LUN. > > > > > > > > > On Fri, Sep 13, 2013 at 10:27 PM, Mike Tutkowski > > > wrote: > > > > OK, yeah, the ACL part will be interesting. That is a bit different > > from > > > how > > > > it works with XenServer and VMware. > > > > > > > > Just to give you an idea how it works in 4.2 with XenServer: > > > > > > > > * The user creates a CS volume (this is just recorded in the > > > cloud.volumes > > > > table). > > > > > > > > * The user attaches the volume as a disk to a VM for the first time > (if > > > the > > > > storage allocator picks the SolidFire plug-in, the storage framewor= k > > > invokes > > > > a method on the plug-in that creates a volume on the SAN...info lik= e > > the > > > IQN > > > > of the SAN volume is recorded in the DB). > > > > > > > > * CitrixResourceBase's execute(AttachVolumeCommand) is executed. It > > > > determines based on a flag passed in that the storage in question i= s > > > > "CloudStack-managed" storage (as opposed to "traditional" > preallocated > > > > storage). This tells it to discover the iSCSI target. Once discover= ed > > it > > > > determines if the iSCSI target already contains a storage repositor= y > > (it > > > > would if this were a re-attach situation). If it does contain an SR > > > already, > > > > then there should already be one VDI, as well. If there is no SR, a= n > SR > > > is > > > > created and a single VDI is created within it (that takes up about = as > > > much > > > > space as was requested for the CloudStack volume). > > > > > > > > * The normal attach-volume logic continues (it depends on the > existence > > > of > > > > an SR and a VDI). > > > > > > > > The VMware case is essentially the same (mainly just substitute > > datastore > > > > for SR and VMDK for VDI). > > > > > > > > In both cases, all hosts in the cluster have discovered the iSCSI > > target, > > > > but only the host that is currently running the VM that is using th= e > > VDI > > > (or > > > > VMKD) is actually using the disk. > > > > > > > > Live Migration should be OK because the hypervisors communicate wit= h > > > > whatever metadata they have on the SR (or datastore). > > > > > > > > I see what you're saying with KVM, though. > > > > > > > > In that case, the hosts are clustered only in CloudStack's eyes. CS > > > controls > > > > Live Migration. You don't really need a clustered filesystem on the > > LUN. > > > The > > > > LUN could be handed over raw to the VM using it. > > > > > > > > If there is a way for me to update the ACL list on the SAN to have > > only a > > > > single KVM host have access to the volume, that would be ideal. > > > > > > > > Also, I agree I'll need to use iscsiadm to discover and log in to t= he > > > iSCSI > > > > target. I'll also need to take the resultant new device and pass it > > into > > > the > > > > VM. > > > > > > > > Does this sound reasonable? Please call me out on anything I seem > > > incorrect > > > > about. :) > > > > > > > > Thanks for all the thought on this, Marcus! > > > > > > > > > > > > On Fri, Sep 13, 2013 at 8:25 PM, Marcus Sorensen < > shadowsor@gmail.com> > > > > wrote: > > > >> > > > >> Perfect. You'll have a domain def ( the VM), a disk def, and the > > attach > > > >> the disk def to the vm. You may need to do your own StorageAdaptor > and > > > run > > > >> iscsiadm commands to accomplish that, depending on how the libvirt > > iscsi > > > >> works. My impression is that a 1:1:1 pool/lun/volume isn't how it > > works > > > on > > > >> xen at the momen., nor is it ideal. > > > >> > > > >> Your plugin will handle acls as far as which host can see which lu= ns > > as > > > >> well, I remember discussing that months ago, so that a disk won't = be > > > >> connected until the hypervisor has exclusive access, so it will be > > safe > > > and > > > >> fence the disk from rogue nodes that cloudstack loses connectivity > > > with. It > > > >> should revoke access to everything but the target host... Except f= or > > > during > > > >> migration but we can discuss that later, there's a migration prep > > > process > > > >> where the new host can be added to the acls, and the old host can = be > > > removed > > > >> post migration. > > > >> > > > >> On Sep 13, 2013 8:16 PM, "Mike Tutkowski" < > > mike.tutkowski@solidfire.com > > > > > > > >> wrote: > > > >>> > > > >>> Yeah, that would be ideal. > > > >>> > > > >>> So, I would still need to discover the iSCSI target, log in to it= , > > then > > > >>> figure out what /dev/sdX was created as a result (and leave it as > is > > - > > > do > > > >>> not format it with any file system...clustered or not). I would > pass > > > that > > > >>> device into the VM. > > > >>> > > > >>> Kind of accurate? > > > >>> > > > >>> > > > >>> On Fri, Sep 13, 2013 at 8:07 PM, Marcus Sorensen < > > shadowsor@gmail.com> > > > >>> wrote: > > > >>>> > > > >>>> Look in LibvirtVMDef.java (I think) for the disk definitions. > There > > > are > > > >>>> ones that work for block devices rather than files. You can pigg= y > > > back off > > > >>>> of the existing disk definitions and attach it to the vm as a > block > > > device. > > > >>>> The definition is an XML string per libvirt XML format. You may > want > > > to use > > > >>>> an alternate path to the disk rather than just /dev/sdx like I > > > mentioned, > > > >>>> there are by-id paths to the block devices, as well as other one= s > > > that will > > > >>>> be consistent and easier for management, not sure how familiar y= ou > > > are with > > > >>>> device naming on Linux. > > > >>>> > > > >>>> On Sep 13, 2013 8:00 PM, "Marcus Sorensen" > > > wrote: > > > >>>>> > > > >>>>> No, as that would rely on virtualized network/iscsi initiator > > inside > > > >>>>> the vm, which also sucks. I mean attach /dev/sdx (your lun on > > > hypervisor) as > > > >>>>> a disk to the VM, rather than attaching some image file that > > resides > > > on a > > > >>>>> filesystem, mounted on the host, living on a target. > > > >>>>> > > > >>>>> Actually, if you plan on the storage supporting live migration = I > > > think > > > >>>>> this is the only way. You can't put a filesystem on it and moun= t > it > > > in two > > > >>>>> places to facilitate migration unless its a clustered filesyste= m, > > in > > > which > > > >>>>> case you're back to shared mount point. > > > >>>>> > > > >>>>> As far as I'm aware, the xenserver SR style is basically LVM > with a > > > xen > > > >>>>> specific cluster management, a custom CLVM. They don't use a > > > filesystem > > > >>>>> either. > > > >>>>> > > > >>>>> On Sep 13, 2013 7:44 PM, "Mike Tutkowski" > > > >>>>> wrote: > > > >>>>>> > > > >>>>>> When you say, "wire up the lun directly to the vm," do you mea= n > > > >>>>>> circumventing the hypervisor? I didn't think we could do that = in > > CS. > > > >>>>>> OpenStack, on the other hand, always circumvents the hyperviso= r, > > as > > > far as I > > > >>>>>> know. > > > >>>>>> > > > >>>>>> > > > >>>>>> On Fri, Sep 13, 2013 at 7:40 PM, Marcus Sorensen < > > > shadowsor@gmail.com> > > > >>>>>> wrote: > > > >>>>>>> > > > >>>>>>> Better to wire up the lun directly to the vm unless there is = a > > good > > > >>>>>>> reason not to. > > > >>>>>>> > > > >>>>>>> On Sep 13, 2013 7:40 PM, "Marcus Sorensen" < > shadowsor@gmail.com> > > > >>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>> You could do that, but as mentioned I think its a mistake to > go > > to > > > >>>>>>>> the trouble of creating a 1:1 mapping of CS volumes to luns > and > > > then putting > > > >>>>>>>> a filesystem on it, mounting it, and then putting a QCOW2 or > > even > > > RAW disk > > > >>>>>>>> image on that filesystem. You'll lose a lot of iops along th= e > > > way, and have > > > >>>>>>>> more overhead with the filesystem and its journaling, etc. > > > >>>>>>>> > > > >>>>>>>> On Sep 13, 2013 7:33 PM, "Mike Tutkowski" > > > >>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> Ah, OK, I didn't know that was such new ground in KVM with > CS. > > > >>>>>>>>> > > > >>>>>>>>> So, the way people use our SAN with KVM and CS today is by > > > >>>>>>>>> selecting SharedMountPoint and specifying the location of t= he > > > share. > > > >>>>>>>>> > > > >>>>>>>>> They can set up their share using Open iSCSI by discovering > > their > > > >>>>>>>>> iSCSI target, logging in to it, then mounting it somewhere = on > > > their file > > > >>>>>>>>> system. > > > >>>>>>>>> > > > >>>>>>>>> Would it make sense for me to just do that discovery, loggi= ng > > in, > > > >>>>>>>>> and mounting behind the scenes for them and letting the > current > > > code manage > > > >>>>>>>>> the rest as it currently does? > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> On Fri, Sep 13, 2013 at 7:27 PM, Marcus Sorensen > > > >>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> Oh, hypervisor snapshots are a bit different. I need to > catch > > up > > > >>>>>>>>>> on the work done in KVM, but this is basically just disk > > > snapshots + memory > > > >>>>>>>>>> dump. I still think disk snapshots would preferably be > handled > > > by the SAN, > > > >>>>>>>>>> and then memory dumps can go to secondary storage or > something > > > else. This is > > > >>>>>>>>>> relatively new ground with CS and KVM, so we will want to > see > > > how others are > > > >>>>>>>>>> planning theirs. > > > >>>>>>>>>> > > > >>>>>>>>>> On Sep 13, 2013 7:20 PM, "Marcus Sorensen" < > > shadowsor@gmail.com > > > > > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Let me back up and say I don't think you'd use a vdi styl= e > on > > > an > > > >>>>>>>>>>> iscsi lun. I think you'd want to treat it as a RAW format= . > > > Otherwise you're > > > >>>>>>>>>>> putting a filesystem on your lun, mounting it, creating a > > > QCOW2 disk image, > > > >>>>>>>>>>> and that seems unnecessary and a performance killer. > > > >>>>>>>>>>> > > > >>>>>>>>>>> So probably attaching the raw iscsi lun as a disk to the > VM, > > > and > > > >>>>>>>>>>> handling snapshots on the San side via the storage plugin > is > > > best. My > > > >>>>>>>>>>> impression from the storage plugin refactor was that ther= e > > was > > > a snapshot > > > >>>>>>>>>>> service that would allow the San to handle snapshots. > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Sep 13, 2013 7:15 PM, "Marcus Sorensen" < > > > shadowsor@gmail.com> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Ideally volume snapshots can be handled by the SAN back > end, > > > if > > > >>>>>>>>>>>> the SAN supports it. The cloudstack mgmt server could ca= ll > > > your plugin for > > > >>>>>>>>>>>> volume snapshot and it would be hypervisor agnostic. As > far > > > as space, that > > > >>>>>>>>>>>> would depend on how your SAN handles it. With ours, we > carve > > > out luns from a > > > >>>>>>>>>>>> pool, and the snapshot spave comes from the pool and is > > > independent of the > > > >>>>>>>>>>>> LUN size the host sees. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Sep 13, 2013 7:10 PM, "Mike Tutkowski" > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Hey Marcus, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> I wonder if the iSCSI storage pool type for libvirt won= 't > > > work > > > >>>>>>>>>>>>> when you take into consideration hypervisor snapshots? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On XenServer, when you take a hypervisor snapshot, the > VDI > > > for > > > >>>>>>>>>>>>> the snapshot is placed on the same storage repository a= s > > the > > > volume is on. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Same idea for VMware, I believe. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> So, what would happen in my case (let's say for XenServ= er > > and > > > >>>>>>>>>>>>> VMware for 4.3 because I don't support hypervisor > snapshots > > > in 4.2) is I'd > > > >>>>>>>>>>>>> make an iSCSI target that is larger than what the user > > > requested for the > > > >>>>>>>>>>>>> CloudStack volume (which is fine because our SAN thinly > > > provisions volumes, > > > >>>>>>>>>>>>> so the space is not actually used unless it needs to be= ). > > > The CloudStack > > > >>>>>>>>>>>>> volume would be the only "object" on the SAN volume > until a > > > hypervisor > > > >>>>>>>>>>>>> snapshot is taken. This snapshot would also reside on t= he > > > SAN volume. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> If this is also how KVM behaves and there is no creatio= n > of > > > >>>>>>>>>>>>> LUNs within an iSCSI target from libvirt (which, even i= f > > > there were support > > > >>>>>>>>>>>>> for this, our SAN currently only allows one LUN per iSC= SI > > > target), then I > > > >>>>>>>>>>>>> don't see how using this model will work. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Perhaps I will have to go enhance the current way this > > works > > > >>>>>>>>>>>>> with DIR? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> What do you think? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Fri, Sep 13, 2013 at 6:28 PM, Mike Tutkowski > > > >>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> That appears to be the way it's used for iSCSI access > > today. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I suppose I could go that route, too, but I might as > well > > > >>>>>>>>>>>>>> leverage what libvirt has for iSCSI instead. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Fri, Sep 13, 2013 at 6:26 PM, Marcus Sorensen > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> To your question about SharedMountPoint, I believe it > > just > > > >>>>>>>>>>>>>>> acts like a > > > >>>>>>>>>>>>>>> 'DIR' storage type or something similar to that. The > > > end-user > > > >>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>> responsible for mounting a file system that all KVM > hosts > > > can > > > >>>>>>>>>>>>>>> access, > > > >>>>>>>>>>>>>>> and CloudStack is oblivious to what is providing the > > > storage. > > > >>>>>>>>>>>>>>> It could > > > >>>>>>>>>>>>>>> be NFS, or OCFS2, or some other clustered filesystem, > > > >>>>>>>>>>>>>>> cloudstack just > > > >>>>>>>>>>>>>>> knows that the provided directory path has VM images. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On Fri, Sep 13, 2013 at 6:23 PM, Marcus Sorensen > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>> > Oh yes, you can use NFS, LVM, and iSCSI all at the > same > > > >>>>>>>>>>>>>>> > time. > > > >>>>>>>>>>>>>>> > Multiples, in fact. > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > On Fri, Sep 13, 2013 at 6:19 PM, Mike Tutkowski > > > >>>>>>>>>>>>>>> > wrote: > > > >>>>>>>>>>>>>>> >> Looks like you can have multiple storage pools: > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> mtutkowski@ubuntu:~$ virsh pool-list > > > >>>>>>>>>>>>>>> >> Name State Autostart > > > >>>>>>>>>>>>>>> >> ----------------------------------------- > > > >>>>>>>>>>>>>>> >> default active yes > > > >>>>>>>>>>>>>>> >> iSCSI active no > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> On Fri, Sep 13, 2013 at 6:12 PM, Mike Tutkowski > > > >>>>>>>>>>>>>>> >> wrote: > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> Reading through the docs you pointed out. > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> I see what you're saying now. > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> You can create an iSCSI (libvirt) storage pool > based > > on > > > >>>>>>>>>>>>>>> >>> an iSCSI target. > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> In my case, the iSCSI target would only have one > LUN, > > > so > > > >>>>>>>>>>>>>>> >>> there would only > > > >>>>>>>>>>>>>>> >>> be one iSCSI (libvirt) storage volume in the > > (libvirt) > > > >>>>>>>>>>>>>>> >>> storage pool. > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> As you say, my plug-in creates and destroys iSCSI > > > >>>>>>>>>>>>>>> >>> targets/LUNs on the > > > >>>>>>>>>>>>>>> >>> SolidFire SAN, so it is not a problem that libvir= t > > does > > > >>>>>>>>>>>>>>> >>> not support > > > >>>>>>>>>>>>>>> >>> creating/deleting iSCSI targets/LUNs. > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> It looks like I need to test this a bit to see if > > > libvirt > > > >>>>>>>>>>>>>>> >>> supports > > > >>>>>>>>>>>>>>> >>> multiple iSCSI storage pools (as you mentioned, > since > > > >>>>>>>>>>>>>>> >>> each one of its > > > >>>>>>>>>>>>>>> >>> storage pools would map to one of my iSCSI > > > targets/LUNs). > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> On Fri, Sep 13, 2013 at 5:58 PM, Mike Tutkowski > > > >>>>>>>>>>>>>>> >>> wrote: > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> LibvirtStoragePoolDef has this type: > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> public enum poolType { > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> ISCSI("iscsi"), NETFS("netfs"), > > > >>>>>>>>>>>>>>> >>>> LOGICAL("logical"), DIR("dir"), > > > >>>>>>>>>>>>>>> >>>> RBD("rbd"); > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> String _poolType; > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> poolType(String poolType) { > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> _poolType =3D poolType; > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> } > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> @Override > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> public String toString() { > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> return _poolType; > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> } > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> } > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> It doesn't look like the iSCSI type is currently > > being > > > >>>>>>>>>>>>>>> >>>> used, but I'm > > > >>>>>>>>>>>>>>> >>>> understanding more what you were getting at. > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> Can you tell me for today (say, 4.2), when someo= ne > > > >>>>>>>>>>>>>>> >>>> selects the > > > >>>>>>>>>>>>>>> >>>> SharedMountPoint option and uses it with iSCSI, = is > > > that > > > >>>>>>>>>>>>>>> >>>> the "netfs" option > > > >>>>>>>>>>>>>>> >>>> above or is that just for NFS? > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> Thanks! > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> On Fri, Sep 13, 2013 at 5:50 PM, Marcus Sorensen > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> wrote: > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> Take a look at this: > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> > > http://libvirt.org/storage.html#StorageBackendISCSI > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> "Volumes must be pre-allocated on the iSCSI > server, > > > and > > > >>>>>>>>>>>>>>> >>>>> cannot be > > > >>>>>>>>>>>>>>> >>>>> created via the libvirt APIs.", which I believe > > your > > > >>>>>>>>>>>>>>> >>>>> plugin will take > > > >>>>>>>>>>>>>>> >>>>> care of. Libvirt just does the work of logging = in > > and > > > >>>>>>>>>>>>>>> >>>>> hooking it up to > > > >>>>>>>>>>>>>>> >>>>> the VM (I believe the Xen api does that work in > the > > > Xen > > > >>>>>>>>>>>>>>> >>>>> stuff). > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> What I'm not sure about is whether this provide= s > a > > > 1:1 > > > >>>>>>>>>>>>>>> >>>>> mapping, or if > > > >>>>>>>>>>>>>>> >>>>> it just allows you to register 1 iscsi device a= s > a > > > >>>>>>>>>>>>>>> >>>>> pool. You may need > > > >>>>>>>>>>>>>>> >>>>> to write some test code or read up a bit more > about > > > >>>>>>>>>>>>>>> >>>>> this. Let us know. > > > >>>>>>>>>>>>>>> >>>>> If it doesn't, you may just have to write your > own > > > >>>>>>>>>>>>>>> >>>>> storage adaptor > > > >>>>>>>>>>>>>>> >>>>> rather than changing LibvirtStorageAdaptor.java= . > > We > > > >>>>>>>>>>>>>>> >>>>> can cross that > > > >>>>>>>>>>>>>>> >>>>> bridge when we get there. > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> As far as interfacing with libvirt, see the jav= a > > > >>>>>>>>>>>>>>> >>>>> bindings doc. > > > >>>>>>>>>>>>>>> >>>>> http://libvirt.org/sources/java/javadoc/Normall= y, > > > >>>>>>>>>>>>>>> >>>>> you'll see a > > > >>>>>>>>>>>>>>> >>>>> connection object be made, then calls made to > that > > > >>>>>>>>>>>>>>> >>>>> 'conn' object. You > > > >>>>>>>>>>>>>>> >>>>> can look at the LibvirtStorageAdaptor to see ho= w > > that > > > >>>>>>>>>>>>>>> >>>>> is done for > > > >>>>>>>>>>>>>>> >>>>> other pool types, and maybe write some test jav= a > > code > > > >>>>>>>>>>>>>>> >>>>> to see if you > > > >>>>>>>>>>>>>>> >>>>> can interface with libvirt and register iscsi > > storage > > > >>>>>>>>>>>>>>> >>>>> pools before you > > > >>>>>>>>>>>>>>> >>>>> get started. > > > >>>>>>>>>>>>>>> >>>>> > > > >>>>>>>>>>>>>>> >>>>> On Fri, Sep 13, 2013 at 5:31 PM, Mike Tutkowski > > > >>>>>>>>>>>>>>> >>>>> wrote: > > > >>>>>>>>>>>>>>> >>>>> > So, Marcus, I need to investigate libvirt mor= e, > > but > > > >>>>>>>>>>>>>>> >>>>> > you figure it > > > >>>>>>>>>>>>>>> >>>>> > supports > > > >>>>>>>>>>>>>>> >>>>> > connecting to/disconnecting from iSCSI target= s, > > > >>>>>>>>>>>>>>> >>>>> > right? > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > On Fri, Sep 13, 2013 at 5:29 PM, Mike Tutkows= ki > > > >>>>>>>>>>>>>>> >>>>> > wrote: > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> OK, thanks, Marcus > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> I am currently looking through some of the > > classes > > > >>>>>>>>>>>>>>> >>>>> >> you pointed out > > > >>>>>>>>>>>>>>> >>>>> >> last > > > >>>>>>>>>>>>>>> >>>>> >> week or so. > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> On Fri, Sep 13, 2013 at 5:26 PM, Marcus > Sorensen > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> wrote: > > > >>>>>>>>>>>>>>> >>>>> >>> > > > >>>>>>>>>>>>>>> >>>>> >>> Yes, my guess is that you will need the isc= si > > > >>>>>>>>>>>>>>> >>>>> >>> initiator utilities > > > >>>>>>>>>>>>>>> >>>>> >>> installed. There should be standard package= s > > for > > > >>>>>>>>>>>>>>> >>>>> >>> any distro. Then > > > >>>>>>>>>>>>>>> >>>>> >>> you'd call > > > >>>>>>>>>>>>>>> >>>>> >>> an agent storage adaptor to do the initiato= r > > > login. > > > >>>>>>>>>>>>>>> >>>>> >>> See the info I > > > >>>>>>>>>>>>>>> >>>>> >>> sent > > > >>>>>>>>>>>>>>> >>>>> >>> previously about LibvirtStorageAdaptor.java > and > > > >>>>>>>>>>>>>>> >>>>> >>> libvirt iscsi > > > >>>>>>>>>>>>>>> >>>>> >>> storage type > > > >>>>>>>>>>>>>>> >>>>> >>> to see if that fits your need. > > > >>>>>>>>>>>>>>> >>>>> >>> > > > >>>>>>>>>>>>>>> >>>>> >>> On Sep 13, 2013 4:55 PM, "Mike Tutkowski" > > > >>>>>>>>>>>>>>> >>>>> >>> > > > >>>>>>>>>>>>>>> >>>>> >>> wrote: > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> Hi, > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> As you may remember, during the 4.2 releas= e > I > > > >>>>>>>>>>>>>>> >>>>> >>>> developed a SolidFire > > > >>>>>>>>>>>>>>> >>>>> >>>> (storage) plug-in for CloudStack. > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> This plug-in was invoked by the storage > > > framework > > > >>>>>>>>>>>>>>> >>>>> >>>> at the necessary > > > >>>>>>>>>>>>>>> >>>>> >>>> times > > > >>>>>>>>>>>>>>> >>>>> >>>> so that I could dynamically create and > delete > > > >>>>>>>>>>>>>>> >>>>> >>>> volumes on the > > > >>>>>>>>>>>>>>> >>>>> >>>> SolidFire SAN > > > >>>>>>>>>>>>>>> >>>>> >>>> (among other activities). > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> This is necessary so I can establish a 1:1 > > > mapping > > > >>>>>>>>>>>>>>> >>>>> >>>> between a > > > >>>>>>>>>>>>>>> >>>>> >>>> CloudStack > > > >>>>>>>>>>>>>>> >>>>> >>>> volume and a SolidFire volume for QoS. > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> In the past, CloudStack always expected th= e > > > admin > > > >>>>>>>>>>>>>>> >>>>> >>>> to create large > > > >>>>>>>>>>>>>>> >>>>> >>>> volumes ahead of time and those volumes > would > > > >>>>>>>>>>>>>>> >>>>> >>>> likely house many > > > >>>>>>>>>>>>>>> >>>>> >>>> root and > > > >>>>>>>>>>>>>>> >>>>> >>>> data disks (which is not QoS friendly). > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> To make this 1:1 mapping scheme work, I > needed > > > to > > > >>>>>>>>>>>>>>> >>>>> >>>> modify logic in > > > >>>>>>>>>>>>>>> >>>>> >>>> the > > > >>>>>>>>>>>>>>> >>>>> >>>> XenServer and VMware plug-ins so they coul= d > > > >>>>>>>>>>>>>>> >>>>> >>>> create/delete storage > > > >>>>>>>>>>>>>>> >>>>> >>>> repositories/datastores as needed. > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> For 4.3 I want to make this happen with KV= M. > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> I'm coming up to speed with how this might > > work > > > on > > > >>>>>>>>>>>>>>> >>>>> >>>> KVM, but I'm > > > >>>>>>>>>>>>>>> >>>>> >>>> still > > > >>>>>>>>>>>>>>> >>>>> >>>> pretty new to KVM. > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> Does anyone familiar with KVM know how I > will > > > need > > > >>>>>>>>>>>>>>> >>>>> >>>> to interact with > > > >>>>>>>>>>>>>>> >>>>> >>>> the > > > >>>>>>>>>>>>>>> >>>>> >>>> iSCSI target? For example, will I have to > > expect > > > >>>>>>>>>>>>>>> >>>>> >>>> Open iSCSI will be > > > >>>>>>>>>>>>>>> >>>>> >>>> installed on the KVM host and use it for > this > > to > > > >>>>>>>>>>>>>>> >>>>> >>>> work? > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> Thanks for any suggestions, > > > >>>>>>>>>>>>>>> >>>>> >>>> Mike > > > >>>>>>>>>>>>>>> >>>>> >>>> > > > >>>>>>>>>>>>>>> >>>>> >>>> -- > > > >>>>>>>>>>>>>>> >>>>> >>>> Mike Tutkowski > > > >>>>>>>>>>>>>>> >>>>> >>>> Senior CloudStack Developer, SolidFire Inc= . > > > >>>>>>>>>>>>>>> >>>>> >>>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >>>>> >>>> o: 303.746.7302 > > > >>>>>>>>>>>>>>> >>>>> >>>> Advancing the way the world uses the cloud= =99 > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> > > > >>>>>>>>>>>>>>> >>>>> >> -- > > > >>>>>>>>>>>>>>> >>>>> >> Mike Tutkowski > > > >>>>>>>>>>>>>>> >>>>> >> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>>> >>>>> >> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >>>>> >> o: 303.746.7302 > > > >>>>>>>>>>>>>>> >>>>> >> Advancing the way the world uses the cloud= =99 > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > > > > >>>>>>>>>>>>>>> >>>>> > -- > > > >>>>>>>>>>>>>>> >>>>> > Mike Tutkowski > > > >>>>>>>>>>>>>>> >>>>> > Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>>> >>>>> > e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >>>>> > o: 303.746.7302 > > > >>>>>>>>>>>>>>> >>>>> > Advancing the way the world uses the cloud=99 > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> > > > >>>>>>>>>>>>>>> >>>> -- > > > >>>>>>>>>>>>>>> >>>> Mike Tutkowski > > > >>>>>>>>>>>>>>> >>>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>>> >>>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >>>> o: 303.746.7302 > > > >>>>>>>>>>>>>>> >>>> Advancing the way the world uses the cloud=99 > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> > > > >>>>>>>>>>>>>>> >>> -- > > > >>>>>>>>>>>>>>> >>> Mike Tutkowski > > > >>>>>>>>>>>>>>> >>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>>> >>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >>> o: 303.746.7302 > > > >>>>>>>>>>>>>>> >>> Advancing the way the world uses the cloud=99 > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> > > > >>>>>>>>>>>>>>> >> -- > > > >>>>>>>>>>>>>>> >> Mike Tutkowski > > > >>>>>>>>>>>>>>> >> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>>> >> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>>> >> o: 303.746.7302 > > > >>>>>>>>>>>>>>> >> Advancing the way the world uses the cloud=99 > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>> Mike Tutkowski > > > >>>>>>>>>>>>>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>>> o: 303.746.7302 > > > >>>>>>>>>>>>>> Advancing the way the world uses the cloud=99 > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> -- > > > >>>>>>>>>>>>> Mike Tutkowski > > > >>>>>>>>>>>>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>>>>>> o: 303.746.7302 > > > >>>>>>>>>>>>> Advancing the way the world uses the cloud=99 > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> -- > > > >>>>>>>>> Mike Tutkowski > > > >>>>>>>>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>>>>> e: mike.tutkowski@solidfire.com > > > >>>>>>>>> o: 303.746.7302 > > > >>>>>>>>> Advancing the way the world uses the cloud=99 > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Mike Tutkowski > > > >>>>>> Senior CloudStack Developer, SolidFire Inc. > > > >>>>>> e: mike.tutkowski@solidfire.com > > > >>>>>> o: 303.746.7302 > > > >>>>>> Advancing the way the world uses the cloud=99 > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> Mike Tutkowski > > > >>> Senior CloudStack Developer, SolidFire Inc. > > > >>> e: mike.tutkowski@solidfire.com > > > >>> o: 303.746.7302 > > > >>> Advancing the way the world uses the cloud=99 > > > > > > > > > > > > > > > > > > > > -- > > > > Mike Tutkowski > > > > Senior CloudStack Developer, SolidFire Inc. > > > > e: mike.tutkowski@solidfire.com > > > > o: 303.746.7302 > > > > Advancing the way the world uses the cloud=99 > > > > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkowski@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud > > *=99* > > > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --089e015366b46d8acc04e69ecfee--