cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Tue, 04 Jun 2013 21:48:43 GMT
Oh, OK, that sounds really good, John.

Thanks and talk to you tomorrow! :)


On Tue, Jun 4, 2013 at 3:42 PM, John Burwell <jburwell@basho.com> wrote:

> Mike,
>
> I am never at a loss for an opinion.  I some thoughts, but want to
> confirm assumptions and ideas against the solidfire, disk_io_throttle,
> and
> object_store branches.  I hope to collect them in a coherent form
> tomorrow (5 June 2013).
>
> Thanks,
> -John
>
>
> On Jun 4, 2013, at 5:29 PM, Mike Tutkowski <mike.tutkowski@solidfire.com>
> wrote:
>
> > "So, in essence, the SolidFire plugin introduces the notion of a managed
> > iSCSI device and provisioned IOPS."
> >
> > Technically, the SolidFire plug-in just introduces the notion of
> > provisioned storage IOPS.
> >
> > The storage framework that leverages the plug-in was incomplete, so I had
> > to try to add in the notion of a managed iSCSI device.
> >
> > I appreciate all the time you've been spending on this. :) Do you have a
> > recommendation as to how we should accomplish what you're looking for?
> >
> > Thanks!
> >
> >
> > On Tue, Jun 4, 2013 at 3:19 PM, John Burwell <jburwell@basho.com> wrote:
> >
> >> Mike,
> >>
> >> So, in essence, the SolidFire plugin introduces the notion of a managed
> >> iSCSI device and provisioned IOPS.
> >>
> >> I want to see a separation of the management capabilities (i.e. can a
> >> device be managed/does an operator want it managed by CloudStack) from
> the
> >> storage protocol.  Ideally, we should end up with a semantic that will
> >> allow any type of storage device to be managed.  I also want to make
> >> progress on decoupling the storage types from the hypervisor
> definitions.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Jun 4, 2013, at 5:13 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> >> wrote:
> >>
> >>> Hi John,
> >>>
> >>> No problem. Answers are below in red.
> >>>
> >>> Thanks!
> >>>
> >>>
> >>> On Tue, Jun 4, 2013 at 2:55 PM, John Burwell <jburwell@basho.com>
> wrote:
> >>>
> >>>> Mike,
> >>>>
> >>>> Could you please answer the following questions for me with regards to
> >> the
> >>>> operation of the SolidFire plugin:
> >>>>
> >>>> What is the cardinality between iSCSI LUNs and SAN volumes?
> >>> Each SAN volume is equivalent to a single LUN (LUN 0).
> >>> 1 SAN volume : 1 LUN
> >>>
> >>>> What is the cardinality between SAN Volumes and CloudStack Volumes?
> >>> 1 SAN volume : 1 CloudStack volume
> >>>
> >>>> Are the LUN(s) created by the management server or externally by the
> >>>> operator?
> >>> When used with the SolidFire plug-in, a SAN volume (same as a SAN LUN)
> is
> >>> created by the management server (via the plug-in) the first time the
> >>> CloudStack volume is attached to a hypervisor.
> >>>
> >>> If you don't want to use the SolidFire plug-in, but still want to use a
> >>> SolidFire volume (LUN), you can do this already today (prior to 4.2).
> The
> >>> admin manually creates the SAN volume and - in this case - multiple VMs
> >> and
> >>> data disks can share this SAN volume. While you can do this today, it
> is
> >>> not useful if you want to enforce storage QoS.
> >>>
> >>>> Are the SAN volumes by the management server or externally by the
> >> operator?
> >>> When the SolidFire plug-in is used, the SAN volumes are completely
> >> managed
> >>> by the management server (via the plug-in). There is no admin
> >> interaction.
> >>> This allows for a 1:1 mapping between a SAN volume and a CloudStack
> >> volume,
> >>> which is necessary for any storage vendor that supports true QoS.
> >>>
> >>>>
> >>>> I would like to clarify how these pieces are related and expected to
> >>>> operate.
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Jun 4, 2013, at 3:46 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com>
> >>>> wrote:
> >>>>
> >>>>> "In particular, how do we ensure that multiple VMs with provisioned
> >> IOPS
> >>>>> won't be cut off by the underlying storage."
> >>>>>
> >>>>> In the storage QoS world, we need to map a single SAN volume (LUN)
> to a
> >>>>> single CloudStack volume. We cannot have multiple CloudStack volumes
> >>>>> sharing a single SAN volume and still guarantee QoS.
> >>>>>
> >>>>> If the user wants to have a single SAN volume house more than one
> >>>>> CloudStack volume, then can do that today without any of my plug-in
> >> code.
> >>>>>
> >>>>>
> >>>>> On Tue, Jun 4, 2013 at 1:43 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com
> >>>>>> wrote:
> >>>>>
> >>>>>> "The administrator will allocate a SAN volume for CloudStack's use
> >> onto
> >>>>>> which CloudStack volumes will be created."
> >>>>>>
> >>>>>> I think we crossed e-mails. :)
> >>>>>>
> >>>>>> Check out my recent e-mail on this.
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jun 4, 2013 at 1:41 PM, John Burwell <jburwell@basho.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Mike,
> >>>>>>>
> >>>>>>> You are coming to the part which concerns me -- concepts from the
> >>>>>>> hypervisor are leaking into storage layer.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> -John
> >>>>>>>
> >>>>>>> On Jun 4, 2013, at 3:35 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> The weird part is that the iSCSI type is today only used (as far
> as
> >> I
> >>>>>>> know)
> >>>>>>>> in regards to XenServer (when you have not PreSetup an SR).
> >>>>>>>>
> >>>>>>>> If you want to use your iSCSI volume from VMware, it uses the vmfs
> >>>> type.
> >>>>>>>>
> >>>>>>>> If you want to use your iSCSI volume from KVM, it uses the
> >>>>>>> SharedMountPoint
> >>>>>>>> type.
> >>>>>>>>
> >>>>>>>> So, I suppose mine and Edison's thinking here was to make a new
> type
> >>>> of
> >>>>>>>> storage to describe this dynamic ability Edison added into the
> >> storage
> >>>>>>>> framework. Maybe it should be more specificy, though:
> Dynamic_iSCSI
> >>>>>>> versus,
> >>>>>>>> say, Dynamic_FC.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Jun 4, 2013 at 1:27 PM, Mike Tutkowski <
> >>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> "The storage device itself shouldn't know or care that it is
> being
> >>>> used
> >>>>>>>>> for a Xen SR -- simply be able to answer questions about it is
> >>>>>>> storing."
> >>>>>>>>>
> >>>>>>>>> I see...so your concern here is that the SolidFire plug-in needs
> to
> >>>>>>> call
> >>>>>>>>> itself "Dynamic" storage so that the hypervisor logic knows to
> >> treat
> >>>>>>> it as
> >>>>>>>>> such.
> >>>>>>>>>
> >>>>>>>>> I'm totally open to removing that constraint and just calling it
> >>>> iSCSI
> >>>>>>> or
> >>>>>>>>> whatever. We would just need a way for the hypervisor attach
> logic
> >> to
> >>>>>>>>> detect this new requirement and perform the necessary activities.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Jun 4, 2013 at 1:24 PM, John Burwell <jburwell@basho.com
> >
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Mike,
> >>>>>>>>>>
> >>>>>>>>>> See my responses in-line.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> -John
> >>>>>>>>>>
> >>>>>>>>>> On Jun 4, 2013, at 3:10 PM, Mike Tutkowski <
> >>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I'm trying to picture this:
> >>>>>>>>>>>
> >>>>>>>>>>> "Finally, while CloudStack may be able to manage a device, an
> >>>>>>> operator
> >>>>>>>>>> may
> >>>>>>>>>>> chose to leave it unmanaged by CloudStack (e.g. the device is
> >>>> shared
> >>>>>>> by
> >>>>>>>>>>> many services, and the operator has chosen to dedicate only a
> >>>> portion
> >>>>>>>>>> of it
> >>>>>>>>>>> to CloudStack).  Does my reasoning make sense?"
> >>>>>>>>>>>
> >>>>>>>>>>> I guess I'm not sure how creating a SAN volume via the plug-in
> >>>>>>> (before
> >>>>>>>>>> an
> >>>>>>>>>>> attach request to the hypervisor) would work unless the
> >> hypervisor
> >>>>>>>>>> consumes
> >>>>>>>>>>> the SAN volume in the form of, say, an SR.
> >>>>>>>>>>
> >>>>>>>>>> My thinking is that, independent of CloudStack, an operator
> >>>> allocates
> >>>>>>> a
> >>>>>>>>>> chunk of  a SAN to CloudStack, and exposes it through a LUN.
>  They
> >>>>>>> simply
> >>>>>>>>>> want to turn control of that LUN over to CloudStack, but not
> allow
> >>>>>>>>>> CloudStack to allocate anymore LUNs.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> As the attach logic stands prior to my changes, we would be
> >> passing
> >>>>>>> in a
> >>>>>>>>>>> SAN volume that does not have the necessary hypervisor support
> >>>> (like
> >>>>>>> an
> >>>>>>>>>> SR)
> >>>>>>>>>>> and the logic will fail.
> >>>>>>>>>>>
> >>>>>>>>>>> Are you thinking we should maybe have the storage framework
> >> itself
> >>>>>>>>>> detect
> >>>>>>>>>>> that such a SAN volume needs support from the hypervisor side
> and
> >>>>>>> have
> >>>>>>>>>> it
> >>>>>>>>>>> call into the agent code specifically to create the SR before
> the
> >>>>>>> attach
> >>>>>>>>>>> logic runs in the agent code?
> >>>>>>>>>>
> >>>>>>>>>> I think the hypervisor management plugin should have a rich
> enough
> >>>>>>>>>> interface to storage to determine available for volume storage.
> >> For
> >>>>>>> Xen,
> >>>>>>>>>> this interface would allow the interrogation of the device to
> >>>>>>> determine the
> >>>>>>>>>> SR is present.   The storage device itself shouldn't know or
> care
> >>>>>>> that it
> >>>>>>>>>> is being used for a Xen SR -- simply be able to answer questions
> >>>>>>> about it
> >>>>>>>>>> is storing.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Jun 4, 2013 at 1:01 PM, Mike Tutkowski <
> >>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> So, the flow is as follows:
> >>>>>>>>>>>>
> >>>>>>>>>>>> * The admin registers the SolidFire driver (which is a type of
> >>>>>>>>>> so-called
> >>>>>>>>>>>> Dynamic storage). Once this is done, a new Primary Storage
> shows
> >>>> up
> >>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>> applicable zone.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * The admin creates a Disk Offering that references the
> storage
> >>>> tag
> >>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>> newly created Primary Storage.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * The end user creates a CloudStack volume. This leads to a
> new
> >>>> row
> >>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>> cloud.volumes table.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * The end user attaches the CloudStack volume to a VM (attach
> >>>> disk).
> >>>>>>>>>> This
> >>>>>>>>>>>> leads to the storage framework calling the plug-in to create a
> >> new
> >>>>>>>>>> volume
> >>>>>>>>>>>> on its storage system (in my case, a SAN). The plug-in also
> >>>> updates
> >>>>>>> the
> >>>>>>>>>>>> cloud.volumes row with applicable data (like the IQN of the
> SAN
> >>>>>>>>>> volume).
> >>>>>>>>>>>> This plug-in code is only invoked if the CloudStack volume is
> in
> >>>> the
> >>>>>>>>>>>> 'Allocated' state. After the attach, the volume will be in the
> >>>>>>> 'Ready'
> >>>>>>>>>>>> state (even after a detach disk) and the plug-in code will not
> >> be
> >>>>>>>>>> called
> >>>>>>>>>>>> again to create this SAN volume.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * The hypervisor-attach logic is run and detects the
> CloudStack
> >>>>>>> volume
> >>>>>>>>>> to
> >>>>>>>>>>>> attach needs "assistance" in the form of a hypervisor data
> >>>> structure
> >>>>>>>>>> (ex.
> >>>>>>>>>>>> an SR on XenServer).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:54 PM, Mike Tutkowski <
> >>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> "To ensure that we are in sync on terminology, volume, in
> these
> >>>>>>>>>>>>> definitions, refers to the physical allocation on the device,
> >>>>>>>>>> correct?"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes...when I say 'volume', I try to mean 'SAN volume'.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> To refer to the 'volume' the end user can make in
> CloudStack, I
> >>>>>>> try to
> >>>>>>>>>>>>> use 'CloudStack volume'.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:50 PM, Mike Tutkowski <
> >>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What you say here may very well make sense, but I'm having a
> >>>> hard
> >>>>>>>>>> time
> >>>>>>>>>>>>>> envisioning it.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Perhaps we should draw Edison in on this conversation as he
> >> was
> >>>>>>> the
> >>>>>>>>>>>>>> initial person to suggest the approach I took.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:42 PM, John Burwell <
> >>>> jburwell@basho.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It feels like we are combining two distinct concepts --
> >> storage
> >>>>>>>>>> device
> >>>>>>>>>>>>>>> management and storage protocols.  In both cases, we are
> >>>>>>>>>> communicating with
> >>>>>>>>>>>>>>> ISCSI, but one allows the system to create/delete volumes
> >>>>>>> (Dynamic)
> >>>>>>>>>> on the
> >>>>>>>>>>>>>>> device while the other requires the volume to be volume to
> be
> >>>>>>>>>> managed
> >>>>>>>>>>>>>>> outside of the CloudStack context.  To ensure that we are
> in
> >>>>>>> sync on
> >>>>>>>>>>>>>>> terminology, volume, in these definitions, refers to the
> >>>> physical
> >>>>>>>>>>>>>>> allocation on the device, correct?  Minimally, we must be
> >> able
> >>>> to
> >>>>>>>>>>>>>>> communicate with a storage device to move bits from one
> place
> >>>> to
> >>>>>>>>>> another,
> >>>>>>>>>>>>>>> read bits, delete bits, etc.  Optionally, a storage device
> >> may
> >>>> be
> >>>>>>>>>> able to
> >>>>>>>>>>>>>>> managed by CloudStack.  Therefore, we can have a unmanaged
> >>>> iSCSI
> >>>>>>>>>> device
> >>>>>>>>>>>>>>> onto which we store a Xen SR, and we can have a managed
> >>>> SolidFire
> >>>>>>>>>> iSCSI
> >>>>>>>>>>>>>>> device on which CloudStack is capable of allocating LUNs
> and
> >>>>>>> storing
> >>>>>>>>>>>>>>> volumes.  Finally, while CloudStack may be able to manage a
> >>>>>>> device,
> >>>>>>>>>> an
> >>>>>>>>>>>>>>> operator may chose to leave it unmanaged by CloudStack
> (e.g.
> >>>> the
> >>>>>>>>>> device is
> >>>>>>>>>>>>>>> shared by many services, and the operator has chosen to
> >>>> dedicate
> >>>>>>>>>> only a
> >>>>>>>>>>>>>>> portion of it to CloudStack).  Does my reasoning make
> sense?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Assuming my thoughts above are reasonable, it seems
> >> appropriate
> >>>>>>> to
> >>>>>>>>>>>>>>> strip the management concerns from StoragePoolType, add the
> >>>>>>> notion
> >>>>>>>>>> of a
> >>>>>>>>>>>>>>> storage device with an attached driver that indicates
> whether
> >>>> or
> >>>>>>>>>> not is
> >>>>>>>>>>>>>>> managed by CloudStack, and establish a abstraction
> >>>> representing a
> >>>>>>>>>> physical
> >>>>>>>>>>>>>>> allocation on a device separate that is associated with a
> >>>> volume.
> >>>>>>>>>> With
> >>>>>>>>>>>>>>> these notions in place, hypervisor drivers can declare
> which
> >>>>>>>>>> protocols they
> >>>>>>>>>>>>>>> support and when they encounter a device managed by
> >> CloudStack,
> >>>>>>>>>> utilize the
> >>>>>>>>>>>>>>> management operations exposed by the driver to automate
> >>>>>>> allocation.
> >>>>>>>>>> If
> >>>>>>>>>>>>>>> these thoughts/concepts make sense, then we can sit down
> and
> >>>>>>> drill
> >>>>>>>>>> down to
> >>>>>>>>>>>>>>> a more detailed design.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Jun 3, 2013, at 5:25 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Here is the difference between the current iSCSI type and
> >> the
> >>>>>>>>>> Dynamic
> >>>>>>>>>>>>>>> type:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> iSCSI type: The admin has to go in and create a Primary
> >>>> Storage
> >>>>>>>>>> based
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> the iSCSI type. At this point in time, the iSCSI volume
> must
> >>>>>>> exist
> >>>>>>>>>> on
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> storage system (it is pre-allocated). Future CloudStack
> >>>> volumes
> >>>>>>> are
> >>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>> as VDIs on the SR that was created behind the scenes.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Dynamic type: The admin has to go in and create Primary
> >>>> Storage
> >>>>>>>>>> based
> >>>>>>>>>>>>>>> on a
> >>>>>>>>>>>>>>>> plug-in that will create and delete volumes on its storage
> >>>>>>> system
> >>>>>>>>>>>>>>>> dynamically (as is enabled via the storage framework).
> When
> >> a
> >>>>>>> user
> >>>>>>>>>>>>>>> wants to
> >>>>>>>>>>>>>>>> attach a CloudStack volume that was created, the framework
> >>>> tells
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> plug-in to create a new volume. After this is done, the
> >> attach
> >>>>>>>>>> logic
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> the hypervisor in question is called. No hypervisor data
> >>>>>>> structure
> >>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>> at this point because the volume was just created. The
> >>>>>>> hypervisor
> >>>>>>>>>> data
> >>>>>>>>>>>>>>>> structure must be created.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:21 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> These are new terms, so I should probably have defined
> them
> >>>> up
> >>>>>>>>>> front
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> you. :)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Static storage: Storage that is pre-allocated (ex. an
> admin
> >>>>>>>>>> creates a
> >>>>>>>>>>>>>>>>> volume on a SAN), then a hypervisor data structure is
> >> created
> >>>>>>> to
> >>>>>>>>>>>>>>> consume
> >>>>>>>>>>>>>>>>> the storage (ex. XenServer SR), then that hypervisor data
> >>>>>>>>>> structure
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> consumed by CloudStack. Disks (VDI) are later placed on
> >> this
> >>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>> data structure as needed. In these cases, the attach
> logic
> >>>>>>> assumes
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> hypervisor data structure is already in place and simply
> >>>>>>> attaches
> >>>>>>>>>>>>>>> the VDI
> >>>>>>>>>>>>>>>>> on the hypervisor data structure to the VM in question.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Dynamic storage: Storage that is not pre-allocated.
> Instead
> >>>> of
> >>>>>>>>>>>>>>>>> pre-existent storage, this could be a SAN (not a volume
> on
> >> a
> >>>>>>> SAN,
> >>>>>>>>>>>>>>> but the
> >>>>>>>>>>>>>>>>> SAN itself). The hypervisor data structure must be
> created
> >>>>>>> when an
> >>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>> volume is performed because these types of volumes have
> not
> >>>>>>> been
> >>>>>>>>>>>>>>> pre-hooked
> >>>>>>>>>>>>>>>>> up to such a hypervisor data structure by an admin. Once
> >> the
> >>>>>>>>>> attach
> >>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>> creates, say, an SR on XenServer for this volume, it
> >> attaches
> >>>>>>> the
> >>>>>>>>>>>>>>> one and
> >>>>>>>>>>>>>>>>> only VDI within the SR to the VM in question.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:13 PM, John Burwell <
> >>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The current implementation of the Dynamic type attach
> >>>> behavior
> >>>>>>>>>>>>>>> works in
> >>>>>>>>>>>>>>>>>> terms of Xen ISCSI which why I ask about the difference.
> >>>>>>> Another
> >>>>>>>>>>>>>>> way to
> >>>>>>>>>>>>>>>>>> ask the question -- what is the definition of a Dynamic
> >>>>>>> storage
> >>>>>>>>>>>>>>> pool type?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 5:10 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> As far as I know, the iSCSI type is uniquely used by
> >>>>>>> XenServer
> >>>>>>>>>>>>>>> when you
> >>>>>>>>>>>>>>>>>>> want to set up Primary Storage that is directly based
> on
> >> an
> >>>>>>>>>> iSCSI
> >>>>>>>>>>>>>>>>>> target.
> >>>>>>>>>>>>>>>>>>> This allows you to skip the step of going to the
> >> hypervisor
> >>>>>>> and
> >>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>>>>>>>> storage repository based on that iSCSI target as
> >> CloudStack
> >>>>>>> does
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>>> for you. I think this is only supported for XenServer.
> >> For
> >>>>>>> all
> >>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> hypervisors, you must first go to the hypervisor and
> >>>> perform
> >>>>>>>>>> this
> >>>>>>>>>>>>>>> step
> >>>>>>>>>>>>>>>>>>> manually.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I don't really know what RBD is.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 2:13 PM, John Burwell <
> >>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Reading through the code, what is the difference
> between
> >>>> the
> >>>>>>>>>>>>>>> ISCSI and
> >>>>>>>>>>>>>>>>>>>> Dynamic types?  Why isn't RBD considered Dynamic?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 3:46 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This new type of storage is defined in the
> >>>>>>>>>>>>>>> Storage.StoragePoolType
> >>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>> (called Dynamic):
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> public static enum StoragePoolType {
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Filesystem(false), // local directory
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> NetworkFilesystem(true), // NFS or CIFS
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> IscsiLUN(true), // shared LUN, with a clusterfs
> >> overlay
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Iscsi(true), // for e.g., ZFS Comstar
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ISO(false), // for iso image
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> LVM(false), // XenServer local LVM SR
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> CLVM(true),
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> RBD(true),
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> SharedMountPoint(true),
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> VMFS(true), // VMware VMFS storage
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> PreSetup(true), // for XenServer, Storage Pool is set
> >>>> up
> >>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> customers.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> EXT(false), // XenServer local EXT SR
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> OCFS2(true),
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Dynamic(true); // dynamic, zone-wide storage (ex.
> >>>>>>>>>> SolidFire)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> boolean shared;
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> StoragePoolType(boolean shared) {
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>     this.shared = shared;
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> public boolean isShared() {
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>     return shared;
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:41 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For example, let's say another storage company wants
> >> to
> >>>>>>>>>>>>>>> implement a
> >>>>>>>>>>>>>>>>>>>>>> plug-in to leverage its Quality of Service feature.
> It
> >>>>>>> would
> >>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> dynamic,
> >>>>>>>>>>>>>>>>>>>>>> zone-wide storage, as well. They would need only
> >>>>>>> implement a
> >>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> plug-in as I've made the necessary changes to the
> >>>>>>>>>>>>>>> hypervisor-attach
> >>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>> to support their plug-in.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:39 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Oh, sorry to imply the XenServer code is SolidFire
> >>>>>>> specific.
> >>>>>>>>>>>>>>> It is
> >>>>>>>>>>>>>>>>>> not.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The XenServer attach logic is now aware of dynamic,
> >>>>>>>>>> zone-wide
> >>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>> (and SolidFire is an implementation of this kind of
> >>>>>>>>>> storage).
> >>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>> kind of
> >>>>>>>>>>>>>>>>>>>>>>> storage is new to 4.2 with Edison's storage
> framework
> >>>>>>>>>> changes.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Edison created a new framework that supported the
> >>>>>>> creation
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> deletion
> >>>>>>>>>>>>>>>>>>>>>>> of volumes dynamically. However, when I visited
> with
> >>>> him
> >>>>>>> in
> >>>>>>>>>>>>>>> Portland
> >>>>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>> in April, we realized that it was not complete. We
> >>>>>>> realized
> >>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>>>> nothing CloudStack could do with these volumes
> unless
> >>>> the
> >>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>>>> changed to recognize this new type of storage and
> >>>> create
> >>>>>>> the
> >>>>>>>>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>>>>>> hypervisor data structure.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:28 PM, John Burwell <
> >>>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> It is generally odd to me that any operation in
> the
> >>>>>>> Storage
> >>>>>>>>>>>>>>> layer
> >>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>> understand or care about details.  I expect to see
> >> the
> >>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>>>>> services
> >>>>>>>>>>>>>>>>>>>>>>>> expose a set of operations that can be
> >> composed/driven
> >>>>>>> by
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>> implementations to allocate space/create
> structures
> >>>> per
> >>>>>>>>>> their
> >>>>>>>>>>>>>>>>>> needs.
> >>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> don't invert this dependency, we are going to end
> >>>> with a
> >>>>>>>>>>>>>>> massive
> >>>>>>>>>>>>>>>>>>>> n-to-n
> >>>>>>>>>>>>>>>>>>>>>>>> problem that will make the system increasingly
> >>>>>>> difficult to
> >>>>>>>>>>>>>>>>>> maintain
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> enhance.  Am I understanding that the Xen specific
> >>>>>>>>>> SolidFire
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> located in the CitrixResourceBase class?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:12 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> To delve into this in a bit more detail:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Prior to 4.2 and aside from one setup method for
> >>>>>>>>>> XenServer,
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> admin
> >>>>>>>>>>>>>>>>>>>>>>>> had
> >>>>>>>>>>>>>>>>>>>>>>>>> to first create a volume on the storage system,
> >> then
> >>>> go
> >>>>>>>>>> into
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>> to set up a data structure to make use of the
> >> volume
> >>>>>>> (ex.
> >>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>> repository on XenServer or a datastore on
> ESX(i)).
> >>>> VMs
> >>>>>>> and
> >>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>> disks
> >>>>>>>>>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>>>>>>> shared this storage system's volume.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> With Edison's new storage framework, storage need
> >> no
> >>>>>>>>>> longer
> >>>>>>>>>>>>>>> be so
> >>>>>>>>>>>>>>>>>>>>>>>> static
> >>>>>>>>>>>>>>>>>>>>>>>>> and you can easily create a 1:1 relationship
> >> between
> >>>> a
> >>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>> system's
> >>>>>>>>>>>>>>>>>>>>>>>>> volume and the VM's data disk (necessary for
> >> storage
> >>>>>>>>>> Quality
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> Service).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> You can now write a plug-in that is called to
> >>>>>>> dynamically
> >>>>>>>>>>>>>>> create
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>> volumes as needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The problem that the storage framework did not
> >>>> address
> >>>>>>> is
> >>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> creating
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> deleting the hypervisor-specific data structure
> >> when
> >>>>>>>>>>>>>>> performing an
> >>>>>>>>>>>>>>>>>>>>>>>>> attach/detach.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> That being the case, I've been enhancing it to do
> >> so.
> >>>>>>> I've
> >>>>>>>>>>>>>>> got
> >>>>>>>>>>>>>>>>>>>>>>>> XenServer
> >>>>>>>>>>>>>>>>>>>>>>>>> worked out and submitted. I've got ESX(i) in my
> >>>> sandbox
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>> submit
> >>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> if we extend the 4.2 freeze date.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Does that help a bit? :)
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:03 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> The storage plug-in - by itself - is hypervisor
> >>>>>>> agnostic.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> The issue is with the volume-attach logic (in
> the
> >>>>>>> agent
> >>>>>>>>>>>>>>> code).
> >>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>> framework calls into the plug-in to have it
> >> create a
> >>>>>>>>>> volume
> >>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>>> needed,
> >>>>>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>>>> when the time comes to attach the volume to a
> >>>>>>> hypervisor,
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>> has to be smart enough to recognize it's being
> >>>>>>> invoked on
> >>>>>>>>>>>>>>>>>> zone-wide
> >>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>> (where the volume has just been created) and
> >> create,
> >>>>>>>>>> say, a
> >>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>> repository (for XenServer) or a datastore (for
> >>>>>>> VMware) to
> >>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>>>>> volume that was just created.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I've been spending most of my time recently
> making
> >>>> the
> >>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>>>>>>>> in the agent code.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Does that clear it up?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 12:48 PM, John Burwell <
> >>>>>>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why the the storage driver is
> >>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>> specific?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 1:21 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, ultimately I would like to support all
> >>>>>>> hypervisors
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> supports. I think I'm just out of time for 4.2
> >> to
> >>>>>>> get
> >>>>>>>>>> KVM
> >>>>>>>>>>>>>>> in.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now this plug-in supports XenServer.
> >>>>>>> Depending on
> >>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> regards to 4.2 feature freeze, I have it
> working
> >>>> for
> >>>>>>>>>>>>>>> VMware in
> >>>>>>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>>>> sandbox,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> as well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, just to be clear, this is all in regards
> >> to
> >>>>>>> Disk
> >>>>>>>>>>>>>>>>>> Offerings.
> >>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>> plan to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> support Compute Offerings post 4.2.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 11:14 AM, Kelcey
> Jamison
> >>>>>>> Damage
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> kelcey@bbits.ca
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any plan on supporting KVM in the
> >> patch
> >>>>>>> cycle
> >>>>>>>>>>>>>>> post
> >>>>>>>>>>>>>>>>>> 4.2?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Mike Tutkowski" <
> >>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Monday, June 3, 2013 10:12:32 AM
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [MERGE] disk_io_throttling to
> >> MASTER
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree on merging Wei's feature first, then
> >>>> mine.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If his feature is for KVM only, then it is a
> >> non
> >>>>>>> issue
> >>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KVM in 4.2.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 8:55 AM, Wei ZHOU <
> >>>>>>>>>>>>>>>>>> ustcweizhou@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the billing, as no one works on billing
> >> now,
> >>>>>>>>>> users
> >>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> calculate
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the billing by themselves. They can get the
> >>>>>>>>>>>>>>> service_offering
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_offering of a VMs and volumes for
> >>>>>>> calculation.
> >>>>>>>>>> Of
> >>>>>>>>>>>>>>> course
> >>>>>>>>>>>>>>>>>>>>>>>> it is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tell user the exact limitation value of
> >>>>>>> individual
> >>>>>>>>>>>>>>> volume,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate limitation for nics as well. I can work
> >> on
> >>>> it
> >>>>>>>>>>>>>>> later. Do
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>>>>>>> think it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a part of I/O throttling?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry my misunstand the second the question.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Agree with what you said about the two
> >> features.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell <jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 2:13 AM, Wei ZHOU <
> >>>>>>>>>>>>>>> ustcweizhou@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John, Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope Mike's aswer helps you. I am trying
> >> to
> >>>>>>>>>> adding
> >>>>>>>>>>>>>>> more.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) I think billing should depend on IO
> >>>>>>> statistics
> >>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation. Please review disk_io_stat if
> >> you
> >>>>>>> have
> >>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_stat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get the IO statistics including bytes/iops
> >>>>>>>>>> read/write
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> individual
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virtual machine.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Going by the AWS model, customers are
> billed
> >>>> more
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> volumes
> >>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioned IOPS, as well as, for those
> >>>>>>> operations (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://aws.amazon.com/ebs/).  I would
> >> imagine
> >>>>>>> our
> >>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> option to employ similar cost models.
>  Could
> >> an
> >>>>>>>>>>>>>>> operator
> >>>>>>>>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> such a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> billing model in the current patch?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) Do you mean IOPS runtime change? KVM
> >>>>>>> supports
> >>>>>>>>>>>>>>> setting
> >>>>>>>>>>>>>>>>>>>>>>>> IOPS/BPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation for a running virtual machine
> >>>> through
> >>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>> line.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CloudStack does not support changing the
> >>>>>>> parameters
> >>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offering
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (computer offering or disk offering).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I meant at the Java interface level.  I
> >>>> apologize
> >>>>>>>>>> for
> >>>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>>>>>>>>>>> unclear.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we more generalize allocation algorithms
> >> with a
> >>>>>>> set
> >>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> interfaces
> >>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> describe the service guarantees provided
> by a
> >>>>>>>>>> resource?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) It is a good question. Maybe it is
> >> better
> >>>> to
> >>>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>>> Mike's
> >>>>>>>>>>>>>>>>>>>>>>>>> patch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling as Mike needs to
> consider
> >>>> the
> >>>>>>>>>>>>>>>>>> limitation in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type, I think.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will expand on my thoughts in a later
> >>>> response
> >>>>>>> to
> >>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>> regarding
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> touch points between these two features.  I
> >>>> think
> >>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will need to be merged before SolidFire,
> but
> >> I
> >>>>>>> think
> >>>>>>>>>>>>>>> we need
> >>>>>>>>>>>>>>>>>>>>>>>> closer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordination between the branches (possibly
> >>>> have
> >>>>>>>>>> have
> >>>>>>>>>>>>>>>>>> solidfire
> >>>>>>>>>>>>>>>>>>>>>>>>> track
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling) to coordinate on this
> >>>> issue.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell <jburwell@basho.com
> >
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The things I want to understand are the
> >>>>>>> following:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Is there value in capturing IOPS
> >> policies
> >>>> be
> >>>>>>>>>>>>>>> captured
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data model (e.g. for billing/usage
> >> purposes,
> >>>>>>>>>>>>>>> expressing
> >>>>>>>>>>>>>>>>>>>>>>>>> offerings).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Should there be a common interface
> model
> >>>> for
> >>>>>>>>>>>>>>> reasoning
> >>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioning at runtime?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) How are conflicting provisioned IOPS
> >>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor and storage device reconciled?
> >> In
> >>>>>>>>>>>>>>> particular,
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> scenario
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is lead to believe (and billed) for more
> >> IOPS
> >>>>>>>>>>>>>>> configured
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> a VM
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage device has been configured to
> >>>> deliver.
> >>>>>>>>>>>>>>> Another
> >>>>>>>>>>>>>>>>>>>>>>>> scenario
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent configuration between a VM
> and a
> >>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>> device at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time, but a later modification to storage
> >>>>>>> device
> >>>>>>>>>>>>>>>>>> introduces
> >>>>>>>>>>>>>>>>>>>>>>>>> logical
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistency.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 2, 2013, at 8:38 PM, Mike
> Tutkowski
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe Wei's feature deals with
> >>>> controlling
> >>>>>>> the
> >>>>>>>>>>>>>>> max
> >>>>>>>>>>>>>>>>>>>>>>>> number of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the hypervisor side.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My feature is focused on controlling IOPS
> >>>> from
> >>>>>>> the
> >>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope that helps. :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Jun 2, 2013 at 6:35 PM, John
> >> Burwell
> >>>> <
> >>>>>>>>>>>>>>>>>>>>>>>> jburwell@basho.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My opinion is that no features should be
> >>>>>>> merged
> >>>>>>>>>>>>>>> until all
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> functional
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues have been resolved and it is
> ready
> >> to
> >>>>>>> turn
> >>>>>>>>>>>>>>> over to
> >>>>>>>>>>>>>>>>>>>>>>>> test.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Until
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total Ops vs discrete read/write ops
> issue
> >>>> is
> >>>>>>>>>>>>>>> addressed
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-reviewed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, I don't think this criteria has
> been
> >>>>>>>>>>>>>>> satisfied.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, how does this work
> >>>> intersect/compliment
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> SolidFire
> >>>>>>>>>>>>>>>>>>>>>>>>> patch
> >>>>>>>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://reviews.apache.org/r/11479/)?
> >> As I
> >>>>>>>>>>>>>>> understand
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also involves provisioned IOPS. I would
> >>>> like
> >>>>>>> to
> >>>>>>>>>>>>>>> ensure
> >>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scenario where provisioned IOPS in KVM
> and
> >>>>>>>>>>>>>>> SolidFire are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incompatible.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU <
> >>>>>>>>>>>>>>>>>> ustcweizhou@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sure. I will change it next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/1 Wido den Hollander <
> >> wido@widodh.nl
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 06/01/2013 08:24 AM, Wei ZHOU wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exactly. I have pushed the features into
> >>>>>>> master.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If anyone object thems for technical
> >> reason
> >>>>>>> till
> >>>>>>>>>>>>>>> Monday,
> >>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> revert
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the sake of clarity I just want to
> >>>> mention
> >>>>>>>>>> again
> >>>>>>>>>>>>>>>>>> that we
> >>>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the total IOps to R/W IOps asap so that
> we
> >>>>>>> never
> >>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only total IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You laid the groundwork for the I/O
> >>>> throttling
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> that's
> >>>>>>>>>>>>>>>>>>>>>>>> great!
> >>>>>>>>>>>>>>>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however prevent that we create legacy
> from
> >>>> day
> >>>>>>>>>> #1.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <
> >>>> wido@widodh.nl>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:59 PM, John Burwell
> >> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 -- this enhancement must to
> discretely
> >>>>>>> support
> >>>>>>>>>>>>>>> read
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how it could be fixed later
> >>>> because
> >>>>>>> I
> >>>>>>>>>>>>>>> don't see
> >>>>>>>>>>>>>>>>>>>>>>>> how we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correctly
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> split total IOPS into read and write.
> >>>>>>>>>> Therefore, we
> >>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> stuck
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total unless/until we decided to break
> >>>>>>> backwards
> >>>>>>>>>>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What Wei meant was merging it into
> master
> >>>> now
> >>>>>>> so
> >>>>>>>>>>>>>>> that it
> >>>>>>>>>>>>>>>>>>>>>>>> will go
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2 branch and add Read / Write IOps
> >> before
> >>>>>>> the
> >>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be released with Read and Write
> >> instead
> >>>>>>> of
> >>>>>>>>>>>>>>> Total
> >>>>>>>>>>>>>>>>>> IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is to make the May 31st feature
> >> freeze
> >>>>>>> date.
> >>>>>>>>>>>>>>> But if
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> window
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (see other threads) then it won't be
> >>>> necessary
> >>>>>>>>>> to do
> >>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I also completely agree that there is no
> >>>>>>>>>> association
> >>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk I/O.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 31, 2013, at 9:51 AM, Wido den
> >>>>>>> Hollander <
> >>>>>>>>>>>>>>>>>>>>>>>> wido@widodh.nl
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:13 PM, Wei ZHOU wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. Good question.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I  thought about at the beginning.
> >> Finally I
> >>>>>>>>>>>>>>> decided to
> >>>>>>>>>>>>>>>>>>>>>>>> ignore
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difference of read and write mainly
> >> because
> >>>>>>> the
> >>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>> throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> did
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> care the difference of sent and received
> >>>>>>> bytes as
> >>>>>>>>>>>>>>> well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That reasoning seems odd. Networking and
> >>>> disk
> >>>>>>> I/O
> >>>>>>>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Disk I/O is much more expensive in most
> >>>>>>>>>> situations
> >>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bandwith.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implementing it will be some copy-paste
> >>>> work.
> >>>>>>> It
> >>>>>>>>>>>>>>> could be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> few days. For the deadline of feature
> >>>> freeze,
> >>>>>>> I
> >>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that , if needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It think it's a feature we can't miss.
> But
> >>>> if
> >>>>>>> it
> >>>>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> window we have to make sure we don't
> >> release
> >>>>>>> with
> >>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>> total
> >>>>>>>>>>>>>>>>>>>>>>>>> IOps
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it in 4.3, that will confuse users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <
> >>>> wido@widodh.nl>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/30/2013 06:03 PM, Wei ZHOU wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to merge disk_io_throttling
> >>>>>>> branch
> >>>>>>>>>> into
> >>>>>>>>>>>>>>>>>> master.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If nobody object, I will merge into
> master
> >>>> in
> >>>>>>> 48
> >>>>>>>>>>>>>>> hours.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The purpose is :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Virtual machines are running on the same
> >>>>>>> storage
> >>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>>>> (local
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> share strage). Because of the rate
> >>>> limitation
> >>>>>>> of
> >>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>>>> (such as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> iops), if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one VM has large disk operation, it may
> >>>> affect
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other VMs running on the same storage
> >>>> device.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is neccesary to set the maximum rate
> >> and
> >>>>>>> limit
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>>>>>>> I/O
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VMs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking at the code I see you make no
> >>>>>>> difference
> >>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Qemu and libvirt support setting both a
> >>>>>>> different
> >>>>>>>>>>>>>>> rate
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps which could benefit a lot of users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's also strange, in the polling side
> you
> >>>>>>>>>> collect
> >>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps, but on the throttling side you
> only
> >> go
> >>>>>>> for
> >>>>>>>>>> a
> >>>>>>>>>>>>>>> global
> >>>>>>>>>>>>>>>>>>>>>>>> value.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write IOps are usually much more
> expensive
> >>>>>>> then
> >>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>> IOps,
> >>>>>>>>>>>>>>>>>>>>>>>> so it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like a valid use-case where that an
> admin
> >>>>>>> would
> >>>>>>>>>> set
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> lower
> >>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps vs Read IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since this only supports KVM at this
> >> point I
> >>>>>>>>>> think
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value to at least have the mechanism in
> >>>> place
> >>>>>>> to
> >>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>> both,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this later would be a lot of work.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If a hypervisor doesn't support setting
> >>>>>>> different
> >>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write you can always sum both up and set
> >>>> that
> >>>>>>> as
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> total
> >>>>>>>>>>>>>>>>>>>>>>>>> limit.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why you implemented it
> >> this
> >>>>>>> way?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The feature includes:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the maximum rate of VMs (in
> >>>>>>>>>> disk_offering,
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> global
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) change the maximum rate of VMs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) limit the disk rate (total bps and
> >> iops)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JIRA ticket:
> >> https://issues.apache.org/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192<ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1192
> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FS (I will update later) :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>
> >> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******
> >>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
> >>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://****
> >>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<
> >>>>>>>>>>>>>>>>>> https://cwiki.
> >>>>>>>>>>>>>>>>>>>>>>>> **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>
> >> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Merge check list :-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Did you check the branch's RAT
> execution
> >>>>>>>>>> success?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Are there new dependencies introduced?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What automated testing (unit and
> >>>>>>> integration)
> >>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> included
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Unit tests are added.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What testing has been done to check
> for
> >>>>>>>>>> potential
> >>>>>>>>>>>>>>>>>>>>>>>> regressions?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the bytes rate and IOPS rate on
> >>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>> UI.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) VM operations, including
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deploy, stop, start, reboot, destroy,
> >>>> expunge.
> >>>>>>>>>>>>>>> migrate,
> >>>>>>>>>>>>>>>>>>>>>>>> restore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) Volume operations, including
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Attach, Detach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To review the code, you can try
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> git diff
> >>>>>>>>>>>>>>> c30057635d04a2396f84c588127d7e******be42e503a7
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> f2e5591b710d04cc86815044f5823e******73a4a58944
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>
> >> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******
> >>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
> >>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> >>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://****
> >>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<
> >>>>>>>>>>>>>>>>>> https://cwiki.
> >>>>>>>>>>>>>>>>>>>>>>>> **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>
> >> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] refs/heads/disk_io_throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/****jira/browse/CLOUDSTACK-1301
> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1301
> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>
> issues.apache.org/****jira/**browse/CLOUDSTACK-2071
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071
> >>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <h**ttps://
> >>>>>>>>>>>>>>>>>> issues.apache.org/jira/**browse/CLOUDSTACK-2071<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (**CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -     VM Disk I/O Throttling)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>> http://solidfire.com/solution/overview/?video=play
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>>>>>>>
> http://solidfire.com/solution/overview/?video=play
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>> cloud<
> >> http://solidfire.com/solution/overview/?video=play
> >>>>>
> >>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>> cloud<
> http://solidfire.com/solution/overview/?video=play
> >>>
> >>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>> *™*
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>> *™*
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Mike Tutkowski*
> >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>> o: 303.746.7302
> >>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>> *™*
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Mike Tutkowski*
> >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>> o: 303.746.7302
> >>>>>>>> Advancing the way the world uses the
> >>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>> *™*
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud<
> >>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>> *™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the
> >>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>> *™*
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the
> >>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>> *™*
> >>
> >>
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud<http://solidfire.com/solution/overview/?video=play>
> > *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message