cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Fri, 07 Jun 2013 02:05:10 GMT
Hi John,

Yeah, when you get a chance, refer to the Google doc I sent to you the
other day to see how the GUI looks for provisioned storage IOPS.

Several months ago, I put this topic out on the e-mail list and we decided
to place the Min, Max, and Burst IOPS in the Add Disk Offering dialog.
Other storage vendors are coming out with QoS, so they should be able to
leverage this GUI going forward (even if they, say, only use Max IOPS).
These fields are optional and can be ignored for storage that does not
support provisioned IOPS. Just like the Disk Size field, the admin can
choose to allow the end user to fill in Min, Max, and Burst IOPS.

I'm OK if we do an either/or model (either Wei's feature or mine, as is
decided by the admin).

I'm not sure what we can do about these two features expressing the speed
in different terms. I've never seen a SAN express the IOPS for QoS in any
way other than total IOPS (i.e. not broken in into read/write IOPS).

Thanks


On Thu, Jun 6, 2013 at 7:16 PM, John Burwell <jburwell@basho.com> wrote:

> Wei,
>
> We have been down the rabbit hole a bit on the Storage/Hypervisor layer
> separation, but we still need to reconcile the behavior of hypervisor
> throttled I/O and storage provisioned IOPS.  I see the following issues
> outstanding:
>
> 1. Hypervisor throttled IOPS are expressed as discrete read/write values
> where as storage provisioned IOPS are expressed as total IOPS.
> 2. How do we handle VMs with throttled IOPS attached to storage volumes
> with provisioned IOPS?
> 3. How should usage data be captured for throttled and provisioned IOPS
> that will permit providers to discriminate these guaranteed operations in
> the event they want to bill for it differently?
> 4. What is the user experience for throttled and provisioned IOPS that
> minimizes confusion of these concepts?
>
> My thinking is that a VM can have either utilize hypervisor throttled IOPS
> or storage provisioned IOPS.  This policy would neatly solve items 1 and 2.
>  Since the two facilities would not be permitted to operate together, they
> do not need to be semantically compatible.  I think item 3 can be resolved
> with an additional flag or two on the usage records.  As for Item 4, I am
> not familiar with how these two enhancements are depicted in the user
> interface.  I think we need to review the UI enhancements for both
> enhancements and ensure they are consistent.
>
> Do these solutions make sense?
>
> Thanks,
> -John
>
> On Jun 6, 2013, at 5:22 PM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
>
> > John and Mike,
> >
> > I was busy working on other issues (CLOUDSTACK-2780/2729,
> > CLOUDSTACK-2856/2857/2865, CLOUDSTACK-2823 , CLOUDSTACK-2875 ) this week.
> > I will start to develop on iops/bps changes tomorrow, and ask for second
> > merge request after testing.
> >
> > -Wei
> >
> >
> > 2013/6/6 Mike Tutkowski <mike.tutkowski@solidfire.com>
> >
> >> I believe I understand where you're going with this, John, and have been
> >> re-working this section of code today.
> >>
> >> I should be able to run it by you tomorrow.
> >>
> >> Thanks for the comments,
> >> Mike
> >>
> >>
> >> On Thu, Jun 6, 2013 at 3:12 PM, John Burwell <jburwell@basho.com>
> wrote:
> >>
> >>> Mike,
> >>>
> >>> See my responses in-line below.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Jun 6, 2013, at 11:09 AM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com>
> >>> wrote:
> >>>
> >>> Hi John,
> >>>
> >>> Thanks for the response.
> >>>
> >>> "I am fine with the VolumeManager determining whether or not a Volume
> is
> >>> managed (i.e. not based on the StoragePoolType, but an actual isManaged
> >>> method), and asking the device driver to allocate resources for the
> >> volume
> >>> if it is managed."
> >>>
> >>> Are you thinking you'd like to see an isManaged() method added to the
> >>> PrimaryDataStoreDriver interface? If it returns true, then the storage
> >>> framework could call the manage() (or whatever name) method (which
> would
> >> be
> >>> new to the PrimaryDataStoreDriver interface, as well) and this would
> call
> >>> into a new method in the hypervisor code to create, say on XenServer,
> an
> >> SR?
> >>>
> >>>
> >>> I would like to see canBeManaged() : boolean on DataStoreDriver.  Since
> >>> the notion of Volumes only pertains to primary storage, I would add a
> >>> allocateStorage and deallocateStorage (Storage is a straw man term --
> >>> something other than volume) methods to
> allocate/create/deallocate/delete
> >>> underlying storage.  To my mind, managed is a mutable property of
> >> DataStore
> >>> which can be enabled if/when the underlying DataStoreDriver can be
> >> managed.
> >>> This approach allows operators to override manageability of devices.
> >>>
> >>> In terms of orchestration/process flow for SR, the Xen plugin would be
> >>> responsible for composing DataStore/Volume methods to create any
> >>> directories or files necessary for the SR.  There should be no
> >> dependencies
> >>> from the Storage to the Hypervisor layer.  As I said earlier, such
> >> circular
> >>> dependencies will lead to a tangled, unmaintainable mess.
> >>>
> >>>
> >>> Just want to make sure I'm on the same page with you.
> >>>
> >>> Thanks again, John
> >>>
> >>>
> >>> On Thu, Jun 6, 2013 at 7:44 AM, John Burwell <jburwell@basho.com>
> wrote:
> >>>
> >>>> Mike,
> >>>>
> >>>> Fundamentally, we can't end up with a Storage layer that supports n
> >>>> devices types with each specific behaviors of m hypervisors.  Such a
> >>>> scenario will create an unmaintainable and untestable beast.
> >> Therefore, my
> >>>> thoughts and recommendations are driven to evolve the Storage layer
> >> towards
> >>>> this separation.
> >>>>
> >>>> I am fine with the VolumeManager determining whether or not a Volume
> is
> >>>> managed (i.e. not based on the StoragePoolType, but an actual
> isManaged
> >>>> method), and asking the device driver to allocate resources for the
> >> volume
> >>>> if it is managed.  Furthermore, the device driver needs to indicate
> >> whether
> >>>> or not it supports management operations.  Finally, I think we need to
> >>>> provide the ability for an administrator to elect to have something
> >> that is
> >>>> manageable be unmanaged (i.e. the driver is capable managing the
> device,
> >>>> but the administrator has elected to leave it unmanaged).
> >>>>
> >>>> Creation of a structure on the volume should be done in the Xen
> >>>> hypervisor module using methods exposed by the Storage layer to
> perform
> >>>> low-level operations (e.g. make directories, create a file, etc).
>  This
> >>>> structure is specific to the operation of the Xen hypervisor, as such,
> >>>> should be confined to its implementation.  From my perspective,
> nothing
> >> in
> >>>> the Storage layer should be concerned with content.  From its
> >> perspective,
> >>>> structure and data are opaque.  It provides the means to query the
> data
> >> to
> >>>> support the interpretation of the content by higher-level layers (e.g.
> >>>> Hypervisors).  To my mind, attach should be a composition of
> operations
> >>>> from the Storage layer that varies based on the Volume storage
> protocol
> >>>> (iSCSI, local file system, NFS, RBD, etc).
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Jun 5, 2013, at 12:25 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com>
> >>>> wrote:
> >>>>
> >>>> Hi John,
> >>>>
> >>>> Alternatively to the way the attach logic is implemented in my patch,
> we
> >>>> could do the following:
> >>>>
> >>>> Leave the attach logic in the agent code alone. In VolumeManagerImpl
> we
> >>>> create an AttachVolumeCommand and send it to the hypervisor. Before
> this
> >>>> command is sent, we could check to see if we're dealing with Dynamic
> (or
> >>>> whatever we want to call it) storage and - if so - send a "Create SR"
> >>>> command to the hypervisor. If this returns OK, we would then proceed
> to
> >> the
> >>>> AttachVolumeCommand, as usual.
> >>>>
> >>>> This way the attach logic remains the same and we just add another
> >>>> command to the agent code that is called for this particular type of
> >>>> storage.
> >>>>
> >>>> What do you think?
> >>>>
> >>>>
> >>>> On Tue, Jun 4, 2013 at 5:42 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com> wrote:
> >>>>
> >>>>> Hey John,
> >>>>>
> >>>>> I created a document for a customer today that outlines how the
> plug-in
> >>>>> works from a user standpoint. This will probably be of use to you, as
> >> well,
> >>>>> as you perform the code review.
> >>>>>
> >>>>> I have shared this document with you (you should have received that
> >>>>> information in a separate e-mail).
> >>>>>
> >>>>> Talk to you later!
> >>>>>
> >>>>>
> >>>>> On Tue, Jun 4, 2013 at 3:48 PM, Mike Tutkowski <
> >>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>
> >>>>>> Oh, OK, that sounds really good, John.
> >>>>>>
> >>>>>> Thanks and talk to you tomorrow! :)
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jun 4, 2013 at 3:42 PM, John Burwell <jburwell@basho.com
> >>> wrote:
> >>>>>>
> >>>>>>> Mike,
> >>>>>>>
> >>>>>>> I am never at a loss for an opinion.  I some thoughts, but want to
> >>>>>>> confirm assumptions and ideas against the solidfire,
> >> disk_io_throttle,
> >>>>>>> and
> >>>>>>> object_store branches.  I hope to collect them in a coherent form
> >>>>>>> tomorrow (5 June 2013).
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> -John
> >>>>>>>
> >>>>>>>
> >>>>>>> On Jun 4, 2013, at 5:29 PM, Mike Tutkowski <
> >>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>
> >>>>>>>> "So, in essence, the SolidFire plugin introduces the notion of a
> >>>>>>> managed
> >>>>>>>> iSCSI device and provisioned IOPS."
> >>>>>>>>
> >>>>>>>> Technically, the SolidFire plug-in just introduces the notion of
> >>>>>>>> provisioned storage IOPS.
> >>>>>>>>
> >>>>>>>> The storage framework that leverages the plug-in was incomplete,
> so
> >>>>>>> I had
> >>>>>>>> to try to add in the notion of a managed iSCSI device.
> >>>>>>>>
> >>>>>>>> I appreciate all the time you've been spending on this. :) Do you
> >>>>>>> have a
> >>>>>>>> recommendation as to how we should accomplish what you're looking
> >>>>>>> for?
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Jun 4, 2013 at 3:19 PM, John Burwell <jburwell@basho.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Mike,
> >>>>>>>>>
> >>>>>>>>> So, in essence, the SolidFire plugin introduces the notion of a
> >>>>>>> managed
> >>>>>>>>> iSCSI device and provisioned IOPS.
> >>>>>>>>>
> >>>>>>>>> I want to see a separation of the management capabilities (i.e.
> >> can
> >>>>>>> a
> >>>>>>>>> device be managed/does an operator want it managed by CloudStack)
> >>>>>>> from the
> >>>>>>>>> storage protocol.  Ideally, we should end up with a semantic that
> >>>>>>> will
> >>>>>>>>> allow any type of storage device to be managed.  I also want to
> >> make
> >>>>>>>>> progress on decoupling the storage types from the hypervisor
> >>>>>>> definitions.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> -John
> >>>>>>>>>
> >>>>>>>>> On Jun 4, 2013, at 5:13 PM, Mike Tutkowski <
> >>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi John,
> >>>>>>>>>>
> >>>>>>>>>> No problem. Answers are below in red.
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jun 4, 2013 at 2:55 PM, John Burwell <
> jburwell@basho.com
> >>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Mike,
> >>>>>>>>>>>
> >>>>>>>>>>> Could you please answer the following questions for me with
> >>>>>>> regards to
> >>>>>>>>> the
> >>>>>>>>>>> operation of the SolidFire plugin:
> >>>>>>>>>>>
> >>>>>>>>>>> What is the cardinality between iSCSI LUNs and SAN volumes?
> >>>>>>>>>> Each SAN volume is equivalent to a single LUN (LUN 0).
> >>>>>>>>>> 1 SAN volume : 1 LUN
> >>>>>>>>>>
> >>>>>>>>>>> What is the cardinality between SAN Volumes and CloudStack
> >>>>>>> Volumes?
> >>>>>>>>>> 1 SAN volume : 1 CloudStack volume
> >>>>>>>>>>
> >>>>>>>>>>> Are the LUN(s) created by the management server or externally
> by
> >>>>>>> the
> >>>>>>>>>>> operator?
> >>>>>>>>>> When used with the SolidFire plug-in, a SAN volume (same as a
> SAN
> >>>>>>> LUN) is
> >>>>>>>>>> created by the management server (via the plug-in) the first
> time
> >>>>>>> the
> >>>>>>>>>> CloudStack volume is attached to a hypervisor.
> >>>>>>>>>>
> >>>>>>>>>> If you don't want to use the SolidFire plug-in, but still want
> to
> >>>>>>> use a
> >>>>>>>>>> SolidFire volume (LUN), you can do this already today (prior to
> >>>>>>> 4.2). The
> >>>>>>>>>> admin manually creates the SAN volume and - in this case -
> >>>>>>> multiple VMs
> >>>>>>>>> and
> >>>>>>>>>> data disks can share this SAN volume. While you can do this
> >> today,
> >>>>>>> it is
> >>>>>>>>>> not useful if you want to enforce storage QoS.
> >>>>>>>>>>
> >>>>>>>>>>> Are the SAN volumes by the management server or externally by
> >> the
> >>>>>>>>> operator?
> >>>>>>>>>> When the SolidFire plug-in is used, the SAN volumes are
> >> completely
> >>>>>>>>> managed
> >>>>>>>>>> by the management server (via the plug-in). There is no admin
> >>>>>>>>> interaction.
> >>>>>>>>>> This allows for a 1:1 mapping between a SAN volume and a
> >> CloudStack
> >>>>>>>>> volume,
> >>>>>>>>>> which is necessary for any storage vendor that supports true
> QoS.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to clarify how these pieces are related and
> >> expected
> >>>>>>> to
> >>>>>>>>>>> operate.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> -John
> >>>>>>>>>>>
> >>>>>>>>>>> On Jun 4, 2013, at 3:46 PM, Mike Tutkowski <
> >>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> "In particular, how do we ensure that multiple VMs with
> >>>>>>> provisioned
> >>>>>>>>> IOPS
> >>>>>>>>>>>> won't be cut off by the underlying storage."
> >>>>>>>>>>>>
> >>>>>>>>>>>> In the storage QoS world, we need to map a single SAN volume
> >>>>>>> (LUN) to a
> >>>>>>>>>>>> single CloudStack volume. We cannot have multiple CloudStack
> >>>>>>> volumes
> >>>>>>>>>>>> sharing a single SAN volume and still guarantee QoS.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If the user wants to have a single SAN volume house more than
> >> one
> >>>>>>>>>>>> CloudStack volume, then can do that today without any of my
> >>>>>>> plug-in
> >>>>>>>>> code.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jun 4, 2013 at 1:43 PM, Mike Tutkowski <
> >>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> "The administrator will allocate a SAN volume for
> CloudStack's
> >>>>>>> use
> >>>>>>>>> onto
> >>>>>>>>>>>>> which CloudStack volumes will be created."
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think we crossed e-mails. :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Check out my recent e-mail on this.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jun 4, 2013 at 1:41 PM, John Burwell <
> >>>>>>> jburwell@basho.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You are coming to the part which concerns me -- concepts
> from
> >>>>>>> the
> >>>>>>>>>>>>>> hypervisor are leaking into storage layer.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Jun 4, 2013, at 3:35 PM, Mike Tutkowski <
> >>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The weird part is that the iSCSI type is today only used
> (as
> >>>>>>> far as
> >>>>>>>>> I
> >>>>>>>>>>>>>> know)
> >>>>>>>>>>>>>>> in regards to XenServer (when you have not PreSetup an SR).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If you want to use your iSCSI volume from VMware, it uses
> >> the
> >>>>>>> vmfs
> >>>>>>>>>>> type.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If you want to use your iSCSI volume from KVM, it uses the
> >>>>>>>>>>>>>> SharedMountPoint
> >>>>>>>>>>>>>>> type.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> So, I suppose mine and Edison's thinking here was to make a
> >>>>>>> new type
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>> storage to describe this dynamic ability Edison added into
> >> the
> >>>>>>>>> storage
> >>>>>>>>>>>>>>> framework. Maybe it should be more specificy, though:
> >>>>>>> Dynamic_iSCSI
> >>>>>>>>>>>>>> versus,
> >>>>>>>>>>>>>>> say, Dynamic_FC.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 1:27 PM, Mike Tutkowski <
> >>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> "The storage device itself shouldn't know or care that it
> >> is
> >>>>>>> being
> >>>>>>>>>>> used
> >>>>>>>>>>>>>>>> for a Xen SR -- simply be able to answer questions about
> it
> >>>>>>> is
> >>>>>>>>>>>>>> storing."
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I see...so your concern here is that the SolidFire plug-in
> >>>>>>> needs to
> >>>>>>>>>>>>>> call
> >>>>>>>>>>>>>>>> itself "Dynamic" storage so that the hypervisor logic
> knows
> >>>>>>> to
> >>>>>>>>> treat
> >>>>>>>>>>>>>> it as
> >>>>>>>>>>>>>>>> such.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm totally open to removing that constraint and just
> >>>>>>> calling it
> >>>>>>>>>>> iSCSI
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>> whatever. We would just need a way for the hypervisor
> >> attach
> >>>>>>> logic
> >>>>>>>>> to
> >>>>>>>>>>>>>>>> detect this new requirement and perform the necessary
> >>>>>>> activities.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 1:24 PM, John Burwell <
> >>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> See my responses in-line.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Jun 4, 2013, at 3:10 PM, Mike Tutkowski <
> >>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'm trying to picture this:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> "Finally, while CloudStack may be able to manage a
> >> device,
> >>>>>>> an
> >>>>>>>>>>>>>> operator
> >>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>> chose to leave it unmanaged by CloudStack (e.g. the
> >> device
> >>>>>>> is
> >>>>>>>>>>> shared
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> many services, and the operator has chosen to dedicate
> >>>>>>> only a
> >>>>>>>>>>> portion
> >>>>>>>>>>>>>>>>> of it
> >>>>>>>>>>>>>>>>>> to CloudStack).  Does my reasoning make sense?"
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I guess I'm not sure how creating a SAN volume via the
> >>>>>>> plug-in
> >>>>>>>>>>>>>> (before
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> attach request to the hypervisor) would work unless the
> >>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>> consumes
> >>>>>>>>>>>>>>>>>> the SAN volume in the form of, say, an SR.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> My thinking is that, independent of CloudStack, an
> >> operator
> >>>>>>>>>>> allocates
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> chunk of  a SAN to CloudStack, and exposes it through a
> >>>>>>> LUN.  They
> >>>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>>> want to turn control of that LUN over to CloudStack, but
> >>>>>>> not allow
> >>>>>>>>>>>>>>>>> CloudStack to allocate anymore LUNs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> As the attach logic stands prior to my changes, we would
> >> be
> >>>>>>>>> passing
> >>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>> SAN volume that does not have the necessary hypervisor
> >>>>>>> support
> >>>>>>>>>>> (like
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> SR)
> >>>>>>>>>>>>>>>>>> and the logic will fail.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Are you thinking we should maybe have the storage
> >> framework
> >>>>>>>>> itself
> >>>>>>>>>>>>>>>>> detect
> >>>>>>>>>>>>>>>>>> that such a SAN volume needs support from the hypervisor
> >>>>>>> side and
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> call into the agent code specifically to create the SR
> >>>>>>> before the
> >>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>> logic runs in the agent code?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think the hypervisor management plugin should have a
> >> rich
> >>>>>>> enough
> >>>>>>>>>>>>>>>>> interface to storage to determine available for volume
> >>>>>>> storage.
> >>>>>>>>> For
> >>>>>>>>>>>>>> Xen,
> >>>>>>>>>>>>>>>>> this interface would allow the interrogation of the
> device
> >>>>>>> to
> >>>>>>>>>>>>>> determine the
> >>>>>>>>>>>>>>>>> SR is present.   The storage device itself shouldn't know
> >>>>>>> or care
> >>>>>>>>>>>>>> that it
> >>>>>>>>>>>>>>>>> is being used for a Xen SR -- simply be able to answer
> >>>>>>> questions
> >>>>>>>>>>>>>> about it
> >>>>>>>>>>>>>>>>> is storing.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 1:01 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> So, the flow is as follows:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> * The admin registers the SolidFire driver (which is a
> >>>>>>> type of
> >>>>>>>>>>>>>>>>> so-called
> >>>>>>>>>>>>>>>>>>> Dynamic storage). Once this is done, a new Primary
> >>>>>>> Storage shows
> >>>>>>>>>>> up
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> applicable zone.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> * The admin creates a Disk Offering that references the
> >>>>>>> storage
> >>>>>>>>>>> tag
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> newly created Primary Storage.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> * The end user creates a CloudStack volume. This leads
> >> to
> >>>>>>> a new
> >>>>>>>>>>> row
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> cloud.volumes table.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> * The end user attaches the CloudStack volume to a VM
> >>>>>>> (attach
> >>>>>>>>>>> disk).
> >>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>> leads to the storage framework calling the plug-in to
> >>>>>>> create a
> >>>>>>>>> new
> >>>>>>>>>>>>>>>>> volume
> >>>>>>>>>>>>>>>>>>> on its storage system (in my case, a SAN). The plug-in
> >>>>>>> also
> >>>>>>>>>>> updates
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> cloud.volumes row with applicable data (like the IQN of
> >>>>>>> the SAN
> >>>>>>>>>>>>>>>>> volume).
> >>>>>>>>>>>>>>>>>>> This plug-in code is only invoked if the CloudStack
> >>>>>>> volume is in
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> 'Allocated' state. After the attach, the volume will be
> >>>>>>> in the
> >>>>>>>>>>>>>> 'Ready'
> >>>>>>>>>>>>>>>>>>> state (even after a detach disk) and the plug-in code
> >>>>>>> will not
> >>>>>>>>> be
> >>>>>>>>>>>>>>>>> called
> >>>>>>>>>>>>>>>>>>> again to create this SAN volume.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> * The hypervisor-attach logic is run and detects the
> >>>>>>> CloudStack
> >>>>>>>>>>>>>> volume
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> attach needs "assistance" in the form of a hypervisor
> >> data
> >>>>>>>>>>> structure
> >>>>>>>>>>>>>>>>> (ex.
> >>>>>>>>>>>>>>>>>>> an SR on XenServer).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:54 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> "To ensure that we are in sync on terminology, volume,
> >>>>>>> in these
> >>>>>>>>>>>>>>>>>>>> definitions, refers to the physical allocation on the
> >>>>>>> device,
> >>>>>>>>>>>>>>>>> correct?"
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Yes...when I say 'volume', I try to mean 'SAN volume'.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> To refer to the 'volume' the end user can make in
> >>>>>>> CloudStack, I
> >>>>>>>>>>>>>> try to
> >>>>>>>>>>>>>>>>>>>> use 'CloudStack volume'.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:50 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What you say here may very well make sense, but I'm
> >>>>>>> having a
> >>>>>>>>>>> hard
> >>>>>>>>>>>>>>>>> time
> >>>>>>>>>>>>>>>>>>>>> envisioning it.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Perhaps we should draw Edison in on this conversation
> >>>>>>> as he
> >>>>>>>>> was
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> initial person to suggest the approach I took.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:42 PM, John Burwell <
> >>>>>>>>>>> jburwell@basho.com
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> It feels like we are combining two distinct concepts
> >> --
> >>>>>>>>> storage
> >>>>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>> management and storage protocols.  In both cases, we
> >>>>>>> are
> >>>>>>>>>>>>>>>>> communicating with
> >>>>>>>>>>>>>>>>>>>>>> ISCSI, but one allows the system to create/delete
> >>>>>>> volumes
> >>>>>>>>>>>>>> (Dynamic)
> >>>>>>>>>>>>>>>>> on the
> >>>>>>>>>>>>>>>>>>>>>> device while the other requires the volume to be
> >>>>>>> volume to be
> >>>>>>>>>>>>>>>>> managed
> >>>>>>>>>>>>>>>>>>>>>> outside of the CloudStack context.  To ensure that
> we
> >>>>>>> are in
> >>>>>>>>>>>>>> sync on
> >>>>>>>>>>>>>>>>>>>>>> terminology, volume, in these definitions, refers to
> >>>>>>> the
> >>>>>>>>>>> physical
> >>>>>>>>>>>>>>>>>>>>>> allocation on the device, correct?  Minimally, we
> >> must
> >>>>>>> be
> >>>>>>>>> able
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> communicate with a storage device to move bits from
> >>>>>>> one place
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> another,
> >>>>>>>>>>>>>>>>>>>>>> read bits, delete bits, etc.  Optionally, a storage
> >>>>>>> device
> >>>>>>>>> may
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> able to
> >>>>>>>>>>>>>>>>>>>>>> managed by CloudStack.  Therefore, we can have a
> >>>>>>> unmanaged
> >>>>>>>>>>> iSCSI
> >>>>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>> onto which we store a Xen SR, and we can have a
> >> managed
> >>>>>>>>>>> SolidFire
> >>>>>>>>>>>>>>>>> iSCSI
> >>>>>>>>>>>>>>>>>>>>>> device on which CloudStack is capable of allocating
> >>>>>>> LUNs and
> >>>>>>>>>>>>>> storing
> >>>>>>>>>>>>>>>>>>>>>> volumes.  Finally, while CloudStack may be able to
> >>>>>>> manage a
> >>>>>>>>>>>>>> device,
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> operator may chose to leave it unmanaged by
> >> CloudStack
> >>>>>>> (e.g.
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> device is
> >>>>>>>>>>>>>>>>>>>>>> shared by many services, and the operator has chosen
> >> to
> >>>>>>>>>>> dedicate
> >>>>>>>>>>>>>>>>> only a
> >>>>>>>>>>>>>>>>>>>>>> portion of it to CloudStack).  Does my reasoning
> make
> >>>>>>> sense?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Assuming my thoughts above are reasonable, it seems
> >>>>>>>>> appropriate
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> strip the management concerns from StoragePoolType,
> >>>>>>> add the
> >>>>>>>>>>>>>> notion
> >>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>> storage device with an attached driver that
> indicates
> >>>>>>> whether
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> not is
> >>>>>>>>>>>>>>>>>>>>>> managed by CloudStack, and establish a abstraction
> >>>>>>>>>>> representing a
> >>>>>>>>>>>>>>>>> physical
> >>>>>>>>>>>>>>>>>>>>>> allocation on a device separate that is associated
> >>>>>>> with a
> >>>>>>>>>>> volume.
> >>>>>>>>>>>>>>>>> With
> >>>>>>>>>>>>>>>>>>>>>> these notions in place, hypervisor drivers can
> >> declare
> >>>>>>> which
> >>>>>>>>>>>>>>>>> protocols they
> >>>>>>>>>>>>>>>>>>>>>> support and when they encounter a device managed by
> >>>>>>>>> CloudStack,
> >>>>>>>>>>>>>>>>> utilize the
> >>>>>>>>>>>>>>>>>>>>>> management operations exposed by the driver to
> >> automate
> >>>>>>>>>>>>>> allocation.
> >>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>> these thoughts/concepts make sense, then we can sit
> >>>>>>> down and
> >>>>>>>>>>>>>> drill
> >>>>>>>>>>>>>>>>> down to
> >>>>>>>>>>>>>>>>>>>>>> a more detailed design.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 5:25 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Here is the difference between the current iSCSI
> >> type
> >>>>>>> and
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>> Dynamic
> >>>>>>>>>>>>>>>>>>>>>> type:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> iSCSI type: The admin has to go in and create a
> >>>>>>> Primary
> >>>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>> the iSCSI type. At this point in time, the iSCSI
> >>>>>>> volume must
> >>>>>>>>>>>>>> exist
> >>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> storage system (it is pre-allocated). Future
> >>>>>>> CloudStack
> >>>>>>>>>>> volumes
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>> as VDIs on the SR that was created behind the
> >> scenes.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Dynamic type: The admin has to go in and create
> >>>>>>> Primary
> >>>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>>>> on a
> >>>>>>>>>>>>>>>>>>>>>>> plug-in that will create and delete volumes on its
> >>>>>>> storage
> >>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>>>>>>>>>> dynamically (as is enabled via the storage
> >>>>>>> framework). When
> >>>>>>>>> a
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>> wants to
> >>>>>>>>>>>>>>>>>>>>>>> attach a CloudStack volume that was created, the
> >>>>>>> framework
> >>>>>>>>>>> tells
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> plug-in to create a new volume. After this is done,
> >>>>>>> the
> >>>>>>>>> attach
> >>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> the hypervisor in question is called. No hypervisor
> >>>>>>> data
> >>>>>>>>>>>>>> structure
> >>>>>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>>>> at this point because the volume was just created.
> >> The
> >>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>> structure must be created.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:21 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> These are new terms, so I should probably have
> >>>>>>> defined them
> >>>>>>>>>>> up
> >>>>>>>>>>>>>>>>> front
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> you. :)
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Static storage: Storage that is pre-allocated (ex.
> >>>>>>> an admin
> >>>>>>>>>>>>>>>>> creates a
> >>>>>>>>>>>>>>>>>>>>>>>> volume on a SAN), then a hypervisor data structure
> >> is
> >>>>>>>>> created
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> consume
> >>>>>>>>>>>>>>>>>>>>>>>> the storage (ex. XenServer SR), then that
> >> hypervisor
> >>>>>>> data
> >>>>>>>>>>>>>>>>> structure
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> consumed by CloudStack. Disks (VDI) are later
> >> placed
> >>>>>>> on
> >>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>> data structure as needed. In these cases, the
> >> attach
> >>>>>>> logic
> >>>>>>>>>>>>>> assumes
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> hypervisor data structure is already in place and
> >>>>>>> simply
> >>>>>>>>>>>>>> attaches
> >>>>>>>>>>>>>>>>>>>>>> the VDI
> >>>>>>>>>>>>>>>>>>>>>>>> on the hypervisor data structure to the VM in
> >>>>>>> question.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Dynamic storage: Storage that is not
> pre-allocated.
> >>>>>>> Instead
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> pre-existent storage, this could be a SAN (not a
> >>>>>>> volume on
> >>>>>>>>> a
> >>>>>>>>>>>>>> SAN,
> >>>>>>>>>>>>>>>>>>>>>> but the
> >>>>>>>>>>>>>>>>>>>>>>>> SAN itself). The hypervisor data structure must be
> >>>>>>> created
> >>>>>>>>>>>>>> when an
> >>>>>>>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>>>> volume is performed because these types of volumes
> >>>>>>> have not
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>> pre-hooked
> >>>>>>>>>>>>>>>>>>>>>>>> up to such a hypervisor data structure by an
> admin.
> >>>>>>> Once
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>> creates, say, an SR on XenServer for this volume,
> >> it
> >>>>>>>>> attaches
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> one and
> >>>>>>>>>>>>>>>>>>>>>>>> only VDI within the SR to the VM in question.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:13 PM, John Burwell <
> >>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The current implementation of the Dynamic type
> >>>>>>> attach
> >>>>>>>>>>> behavior
> >>>>>>>>>>>>>>>>>>>>>> works in
> >>>>>>>>>>>>>>>>>>>>>>>>> terms of Xen ISCSI which why I ask about the
> >>>>>>> difference.
> >>>>>>>>>>>>>> Another
> >>>>>>>>>>>>>>>>>>>>>> way to
> >>>>>>>>>>>>>>>>>>>>>>>>> ask the question -- what is the definition of a
> >>>>>>> Dynamic
> >>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> pool type?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 5:10 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> As far as I know, the iSCSI type is uniquely
> used
> >>>>>>> by
> >>>>>>>>>>>>>> XenServer
> >>>>>>>>>>>>>>>>>>>>>> when you
> >>>>>>>>>>>>>>>>>>>>>>>>>> want to set up Primary Storage that is directly
> >>>>>>> based on
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>> iSCSI
> >>>>>>>>>>>>>>>>>>>>>>>>> target.
> >>>>>>>>>>>>>>>>>>>>>>>>>> This allows you to skip the step of going to the
> >>>>>>>>> hypervisor
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> creating a
> >>>>>>>>>>>>>>>>>>>>>>>>>> storage repository based on that iSCSI target as
> >>>>>>>>> CloudStack
> >>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>>>>>>>>>> for you. I think this is only supported for
> >>>>>>> XenServer.
> >>>>>>>>> For
> >>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>> hypervisors, you must first go to the hypervisor
> >>>>>>> and
> >>>>>>>>>>> perform
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> step
> >>>>>>>>>>>>>>>>>>>>>>>>>> manually.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I don't really know what RBD is.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 2:13 PM, John Burwell <
> >>>>>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Reading through the code, what is the
> difference
> >>>>>>> between
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> ISCSI and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic types?  Why isn't RBD considered
> >> Dynamic?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 3:46 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This new type of storage is defined in the
> >>>>>>>>>>>>>>>>>>>>>> Storage.StoragePoolType
> >>>>>>>>>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (called Dynamic):
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> public static enum StoragePoolType {
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Filesystem(false), // local directory
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetworkFilesystem(true), // NFS or CIFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> IscsiLUN(true), // shared LUN, with a
> clusterfs
> >>>>>>>>> overlay
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Iscsi(true), // for e.g., ZFS Comstar
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ISO(false), // for iso image
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> LVM(false), // XenServer local LVM SR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> CLVM(true),
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> RBD(true),
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> SharedMountPoint(true),
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> VMFS(true), // VMware VMFS storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PreSetup(true), // for XenServer, Storage Pool
> >>>>>>> is set
> >>>>>>>>>>> up
> >>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> customers.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> EXT(false), // XenServer local EXT SR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> OCFS2(true),
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic(true); // dynamic, zone-wide storage
> >> (ex.
> >>>>>>>>>>>>>>>>> SolidFire)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean shared;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> StoragePoolType(boolean shared) {
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>    this.shared = shared;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> public boolean isShared() {
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>    return shared;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:41 PM, Mike Tutkowski
> >> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, let's say another storage
> company
> >>>>>>> wants
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> implement a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> plug-in to leverage its Quality of Service
> >>>>>>> feature. It
> >>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> dynamic,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone-wide storage, as well. They would need
> >> only
> >>>>>>>>>>>>>> implement a
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> plug-in as I've made the necessary changes to
> >>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> hypervisor-attach
> >>>>>>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to support their plug-in.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:39 PM, Mike
> >> Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Oh, sorry to imply the XenServer code is
> >>>>>>> SolidFire
> >>>>>>>>>>>>>> specific.
> >>>>>>>>>>>>>>>>>>>>>> It is
> >>>>>>>>>>>>>>>>>>>>>>>>> not.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The XenServer attach logic is now aware of
> >>>>>>> dynamic,
> >>>>>>>>>>>>>>>>> zone-wide
> >>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (and SolidFire is an implementation of this
> >>>>>>> kind of
> >>>>>>>>>>>>>>>>> storage).
> >>>>>>>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>>>>>>>> kind of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage is new to 4.2 with Edison's storage
> >>>>>>> framework
> >>>>>>>>>>>>>>>>> changes.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Edison created a new framework that
> supported
> >>>>>>> the
> >>>>>>>>>>>>>> creation
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> deletion
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of volumes dynamically. However, when I
> >>>>>>> visited with
> >>>>>>>>>>> him
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> Portland
> >>>>>>>>>>>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in April, we realized that it was not
> >>>>>>> complete. We
> >>>>>>>>>>>>>> realized
> >>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nothing CloudStack could do with these
> >> volumes
> >>>>>>> unless
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed to recognize this new type of
> storage
> >>>>>>> and
> >>>>>>>>>>> create
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor data structure.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:28 PM, John Burwell
> >> <
> >>>>>>>>>>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is generally odd to me that any
> operation
> >>>>>>> in the
> >>>>>>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>>>>>>> layer
> >>>>>>>>>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand or care about details.  I expect
> >>>>>>> to see
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>> Storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>> services
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expose a set of operations that can be
> >>>>>>>>> composed/driven
> >>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementations to allocate space/create
> >>>>>>> structures
> >>>>>>>>>>> per
> >>>>>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>>>>>>>>> needs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't invert this dependency, we are going
> >> to
> >>>>>>> end
> >>>>>>>>>>> with a
> >>>>>>>>>>>>>>>>>>>>>> massive
> >>>>>>>>>>>>>>>>>>>>>>>>>>> n-to-n
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem that will make the system
> >> increasingly
> >>>>>>>>>>>>>> difficult to
> >>>>>>>>>>>>>>>>>>>>>>>>> maintain
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enhance.  Am I understanding that the Xen
> >>>>>>> specific
> >>>>>>>>>>>>>>>>> SolidFire
> >>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> located in the CitrixResourceBase class?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:12 PM, Mike
> >>>>>>> Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To delve into this in a bit more detail:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Prior to 4.2 and aside from one setup
> >> method
> >>>>>>> for
> >>>>>>>>>>>>>>>>> XenServer,
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> admin
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> had
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to first create a volume on the storage
> >>>>>>> system,
> >>>>>>>>> then
> >>>>>>>>>>> go
> >>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to set up a data structure to make use of
> >> the
> >>>>>>>>> volume
> >>>>>>>>>>>>>> (ex.
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repository on XenServer or a datastore on
> >>>>>>> ESX(i)).
> >>>>>>>>>>> VMs
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>>> disks
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shared this storage system's volume.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> With Edison's new storage framework,
> >> storage
> >>>>>>> need
> >>>>>>>>> no
> >>>>>>>>>>>>>>>>> longer
> >>>>>>>>>>>>>>>>>>>>>> be so
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> static
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and you can easily create a 1:1
> >> relationship
> >>>>>>>>> between
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>> system's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> volume and the VM's data disk (necessary
> >> for
> >>>>>>>>> storage
> >>>>>>>>>>>>>>>>> Quality
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Service).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You can now write a plug-in that is called
> >> to
> >>>>>>>>>>>>>> dynamically
> >>>>>>>>>>>>>>>>>>>>>> create
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> volumes as needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The problem that the storage framework did
> >>>>>>> not
> >>>>>>>>>>> address
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> creating
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deleting the hypervisor-specific data
> >>>>>>> structure
> >>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>> performing an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attach/detach.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That being the case, I've been enhancing
> it
> >>>>>>> to do
> >>>>>>>>> so.
> >>>>>>>>>>>>>> I've
> >>>>>>>>>>>>>>>>>>>>>> got
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> XenServer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worked out and submitted. I've got ESX(i)
> >> in
> >>>>>>> my
> >>>>>>>>>>> sandbox
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>> submit
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if we extend the 4.2 freeze date.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does that help a bit? :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:03 PM, Mike
> >>>>>>> Tutkowski <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The storage plug-in - by itself - is
> >>>>>>> hypervisor
> >>>>>>>>>>>>>> agnostic.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The issue is with the volume-attach logic
> >>>>>>> (in the
> >>>>>>>>>>>>>> agent
> >>>>>>>>>>>>>>>>>>>>>> code).
> >>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> framework calls into the plug-in to have
> >> it
> >>>>>>>>> create a
> >>>>>>>>>>>>>>>>> volume
> >>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> needed,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when the time comes to attach the volume
> >> to
> >>>>>>> a
> >>>>>>>>>>>>>> hypervisor,
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has to be smart enough to recognize it's
> >>>>>>> being
> >>>>>>>>>>>>>> invoked on
> >>>>>>>>>>>>>>>>>>>>>>>>> zone-wide
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (where the volume has just been created)
> >> and
> >>>>>>>>> create,
> >>>>>>>>>>>>>>>>> say, a
> >>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repository (for XenServer) or a datastore
> >>>>>>> (for
> >>>>>>>>>>>>>> VMware) to
> >>>>>>>>>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> volume that was just created.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've been spending most of my time
> >> recently
> >>>>>>> making
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> attach
> >>>>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the agent code.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does that clear it up?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 12:48 PM, John
> >>>>>>> Burwell <
> >>>>>>>>>>>>>>>>>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why the the storage
> >> driver
> >>>>>>> is
> >>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>> specific?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 1:21 PM, Mike
> >> Tutkowski
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, ultimately I would like to support
> >>>>>>> all
> >>>>>>>>>>>>>> hypervisors
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports. I think I'm just out of time
> >>>>>>> for 4.2
> >>>>>>>>> to
> >>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>>>> KVM
> >>>>>>>>>>>>>>>>>>>>>> in.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now this plug-in supports
> >> XenServer.
> >>>>>>>>>>>>>> Depending on
> >>>>>>>>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>>>>>>> we do
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards to 4.2 feature freeze, I have
> it
> >>>>>>> working
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> VMware in
> >>>>>>>>>>>>>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sandbox,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, just to be clear, this is all in
> >>>>>>> regards
> >>>>>>>>> to
> >>>>>>>>>>>>>> Disk
> >>>>>>>>>>>>>>>>>>>>>>>>> Offerings.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> plan to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support Compute Offerings post 4.2.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 11:14 AM, Kelcey
> >>>>>>> Jamison
> >>>>>>>>>>>>>> Damage
> >>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kelcey@bbits.ca
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any plan on supporting KVM in
> >>>>>>> the
> >>>>>>>>> patch
> >>>>>>>>>>>>>> cycle
> >>>>>>>>>>>>>>>>>>>>>> post
> >>>>>>>>>>>>>>>>>>>>>>>>> 4.2?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Mike Tutkowski" <
> >>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Monday, June 3, 2013 10:12:32 AM
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [MERGE]
> disk_io_throttling
> >>>>>>> to
> >>>>>>>>> MASTER
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree on merging Wei's feature
> first,
> >>>>>>> then
> >>>>>>>>>>> mine.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If his feature is for KVM only, then
> it
> >>>>>>> is a
> >>>>>>>>> non
> >>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KVM in 4.2.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 8:55 AM, Wei
> >> ZHOU
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> ustcweizhou@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the billing, as no one works on
> >>>>>>> billing
> >>>>>>>>> now,
> >>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> calculate
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the billing by themselves. They can
> >> get
> >>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> service_offering
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_offering of a VMs and volumes
> for
> >>>>>>>>>>>>>> calculation.
> >>>>>>>>>>>>>>>>> Of
> >>>>>>>>>>>>>>>>>>>>>> course
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tell user the exact limitation
> >> value
> >>>>>>> of
> >>>>>>>>>>>>>> individual
> >>>>>>>>>>>>>>>>>>>>>> volume,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rate limitation for nics as well. I
> >> can
> >>>>>>> work
> >>>>>>>>> on
> >>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> later. Do
> >>>>>>>>>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a part of I/O throttling?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry my misunstand the second the
> >>>>>>> question.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Agree with what you said about the
> two
> >>>>>>>>> features.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell <
> >>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 2:13 AM, Wei ZHOU
> >> <
> >>>>>>>>>>>>>>>>>>>>>> ustcweizhou@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John, Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope Mike's aswer helps you. I am
> >>>>>>> trying
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>> adding
> >>>>>>>>>>>>>>>>>>>>>> more.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) I think billing should depend
> on
> >>>>>>> IO
> >>>>>>>>>>>>>> statistics
> >>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation. Please review
> >>>>>>> disk_io_stat if
> >>>>>>>>> you
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_stat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get the IO statistics including
> >>>>>>> bytes/iops
> >>>>>>>>>>>>>>>>> read/write
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> individual
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virtual machine.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Going by the AWS model, customers
> are
> >>>>>>> billed
> >>>>>>>>>>> more
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> volumes
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioned IOPS, as well as, for
> >> those
> >>>>>>>>>>>>>> operations (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://aws.amazon.com/ebs/).  I
> >> would
> >>>>>>>>> imagine
> >>>>>>>>>>>>>> our
> >>>>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> option to employ similar cost
> models.
> >>>>>>> Could
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> operator
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> billing model in the current patch?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) Do you mean IOPS runtime
> change?
> >>>>>>> KVM
> >>>>>>>>>>>>>> supports
> >>>>>>>>>>>>>>>>>>>>>> setting
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS/BPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation for a running virtual
> >>>>>>> machine
> >>>>>>>>>>> through
> >>>>>>>>>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>>>>>>>> line.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CloudStack does not support
> changing
> >>>>>>> the
> >>>>>>>>>>>>>> parameters
> >>>>>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offering
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (computer offering or disk
> >> offering).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I meant at the Java interface level.
> >> I
> >>>>>>>>>>> apologize
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unclear.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we more generalize allocation
> >>>>>>> algorithms
> >>>>>>>>> with a
> >>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> describe the service guarantees
> >>>>>>> provided by a
> >>>>>>>>>>>>>>>>> resource?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) It is a good question. Maybe it
> >> is
> >>>>>>>>> better
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> commit
> >>>>>>>>>>>>>>>>>>>>>>>>> Mike's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling as Mike needs to
> >>>>>>> consider
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> limitation in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type, I think.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will expand on my thoughts in a
> >> later
> >>>>>>>>>>> response
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> touch points between these two
> >>>>>>> features.  I
> >>>>>>>>>>> think
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will need to be merged before
> >>>>>>> SolidFire, but
> >>>>>>>>> I
> >>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>> we need
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordination between the branches
> >>>>>>> (possibly
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>>> solidfire
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> track
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling) to coordinate on
> >>>>>>> this
> >>>>>>>>>>> issue.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell <
> >>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The things I want to understand
> are
> >>>>>>> the
> >>>>>>>>>>>>>> following:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Is there value in capturing
> IOPS
> >>>>>>>>> policies
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> captured
> >>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data model (e.g. for billing/usage
> >>>>>>>>> purposes,
> >>>>>>>>>>>>>>>>>>>>>> expressing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offerings).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Should there be a common
> >>>>>>> interface model
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> reasoning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioning at runtime?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) How are conflicting provisioned
> >>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor and storage device
> >>>>>>> reconciled?
> >>>>>>>>> In
> >>>>>>>>>>>>>>>>>>>>>> particular,
> >>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scenario
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is lead to believe (and billed)
> for
> >>>>>>> more
> >>>>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>> configured
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a VM
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage device has been configured
> >> to
> >>>>>>>>>>> deliver.
> >>>>>>>>>>>>>>>>>>>>>> Another
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scenario
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent configuration between a
> >>>>>>> VM and a
> >>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>> device at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time, but a later modification to
> >>>>>>> storage
> >>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>>>>> introduces
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> logical
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistency.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 2, 2013, at 8:38 PM, Mike
> >>>>>>> Tutkowski
> >>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe Wei's feature deals with
> >>>>>>>>>>> controlling
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> max
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> number of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the hypervisor side.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My feature is focused on
> >> controlling
> >>>>>>> IOPS
> >>>>>>>>>>> from
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> side.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope that helps. :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Jun 2, 2013 at 6:35 PM,
> >> John
> >>>>>>>>> Burwell
> >>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jburwell@basho.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My opinion is that no features
> >>>>>>> should be
> >>>>>>>>>>>>>> merged
> >>>>>>>>>>>>>>>>>>>>>> until all
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functional
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues have been resolved and it
> >> is
> >>>>>>> ready
> >>>>>>>>> to
> >>>>>>>>>>>>>> turn
> >>>>>>>>>>>>>>>>>>>>>> over to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> test.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Until
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total Ops vs discrete read/write
> >>>>>>> ops issue
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> addressed
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> re-reviewed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, I don't think this criteria
> >>>>>>> has been
> >>>>>>>>>>>>>>>>>>>>>> satisfied.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, how does this work
> >>>>>>>>>>> intersect/compliment
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> SolidFire
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> https://reviews.apache.org/r/11479/
> >>>>>>> )?
> >>>>>>>>> As I
> >>>>>>>>>>>>>>>>>>>>>> understand
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also involves provisioned IOPS. I
> >>>>>>> would
> >>>>>>>>>>> like
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> ensure
> >>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scenario where provisioned IOPS
> in
> >>>>>>> KVM and
> >>>>>>>>>>>>>>>>>>>>>> SolidFire are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incompatible.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 1, 2013, at 6:47 AM, Wei
> >>>>>>> ZHOU <
> >>>>>>>>>>>>>>>>>>>>>>>>> ustcweizhou@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sure. I will change it next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/1 Wido den Hollander <
> >>>>>>>>> wido@widodh.nl
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 06/01/2013 08:24 AM, Wei ZHOU
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exactly. I have pushed the
> >> features
> >>>>>>> into
> >>>>>>>>>>>>>> master.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If anyone object thems for
> >> technical
> >>>>>>>>> reason
> >>>>>>>>>>>>>> till
> >>>>>>>>>>>>>>>>>>>>>> Monday,
> >>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> revert
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the sake of clarity I just
> >> want
> >>>>>>> to
> >>>>>>>>>>> mention
> >>>>>>>>>>>>>>>>> again
> >>>>>>>>>>>>>>>>>>>>>>>>> that we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the total IOps to R/W IOps asap
> so
> >>>>>>> that we
> >>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only total IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You laid the groundwork for the
> >> I/O
> >>>>>>>>>>> throttling
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> that's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however prevent that we create
> >>>>>>> legacy from
> >>>>>>>>>>> day
> >>>>>>>>>>>>>>>>> #1.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <
> >>>>>>>>>>> wido@widodh.nl>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:59 PM, John
> >> Burwell
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 -- this enhancement must to
> >>>>>>> discretely
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>> read
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how it could be fixed
> >>>>>>> later
> >>>>>>>>>>> because
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>> don't see
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correctly
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> split total IOPS into read and
> >>>>>>> write.
> >>>>>>>>>>>>>>>>> Therefore, we
> >>>>>>>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stuck
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total unless/until we decided to
> >>>>>>> break
> >>>>>>>>>>>>>> backwards
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What Wei meant was merging it
> into
> >>>>>>> master
> >>>>>>>>>>> now
> >>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>> that it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will go
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2 branch and add Read / Write
> >> IOps
> >>>>>>>>> before
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be released with Read and
> >> Write
> >>>>>>>>> instead
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> Total
> >>>>>>>>>>>>>>>>>>>>>>>>> IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is to make the May 31st
> >> feature
> >>>>>>>>> freeze
> >>>>>>>>>>>>>> date.
> >>>>>>>>>>>>>>>>>>>>>> But if
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> window
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (see other threads) then it won't
> >> be
> >>>>>>>>>>> necessary
> >>>>>>>>>>>>>>>>> to do
> >>>>>>>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I also completely agree that
> there
> >>>>>>> is no
> >>>>>>>>>>>>>>>>> association
> >>>>>>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk I/O.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 31, 2013, at 9:51 AM, Wido
> >>>>>>> den
> >>>>>>>>>>>>>> Hollander <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wido@widodh.nl
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:13 PM, Wei ZHOU
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wido,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. Good question.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I  thought about at the
> beginning.
> >>>>>>>>> Finally I
> >>>>>>>>>>>>>>>>>>>>>> decided to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ignore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difference of read and write
> >> mainly
> >>>>>>>>> because
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> did
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> care the difference of sent and
> >>>>>>> received
> >>>>>>>>>>>>>> bytes as
> >>>>>>>>>>>>>>>>>>>>>> well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That reasoning seems odd.
> >>>>>>> Networking and
> >>>>>>>>>>> disk
> >>>>>>>>>>>>>> I/O
> >>>>>>>>>>>>>>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Disk I/O is much more expensive
> in
> >>>>>>> most
> >>>>>>>>>>>>>>>>> situations
> >>>>>>>>>>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> network
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bandwith.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implementing it will be some
> >>>>>>> copy-paste
> >>>>>>>>>>> work.
> >>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>> could be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> few days. For the deadline of
> >>>>>>> feature
> >>>>>>>>>>> freeze,
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that , if needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It think it's a feature we can't
> >>>>>>> miss. But
> >>>>>>>>>>> if
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> window we have to make sure we
> >> don't
> >>>>>>>>> release
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>>> total
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it in 4.3, that will confuse
> >> users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander <
> >>>>>>>>>>> wido@widodh.nl>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/30/2013 06:03 PM, Wei ZHOU
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to merge
> >>>>>>> disk_io_throttling
> >>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>>> master.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If nobody object, I will merge
> >> into
> >>>>>>> master
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> 48
> >>>>>>>>>>>>>>>>>>>>>> hours.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The purpose is :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Virtual machines are running on
> >> the
> >>>>>>> same
> >>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (local
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> share strage). Because of the
> rate
> >>>>>>>>>>> limitation
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> device
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (such as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> iops), if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one VM has large disk operation,
> >> it
> >>>>>>> may
> >>>>>>>>>>> affect
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other VMs running on the same
> >>>>>>> storage
> >>>>>>>>>>> device.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is neccesary to set the
> maximum
> >>>>>>> rate
> >>>>>>>>> and
> >>>>>>>>>>>>>> limit
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> disk
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I/O
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VMs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking at the code I see you
> make
> >>>>>>> no
> >>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Qemu and libvirt support setting
> >>>>>>> both a
> >>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>> rate
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps which could benefit a lot of
> >>>>>>> users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's also strange, in the polling
> >>>>>>> side you
> >>>>>>>>>>>>>>>>> collect
> >>>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps, but on the throttling side
> >>>>>>> you only
> >>>>>>>>> go
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> global
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write IOps are usually much more
> >>>>>>> expensive
> >>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>>> Read
> >>>>>>>>>>>>>>>>>>>>>>>>> IOps,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like a valid use-case where that
> >> an
> >>>>>>> admin
> >>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> lower
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps vs Read IOps.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since this only supports KVM at
> >> this
> >>>>>>>>> point I
> >>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value to at least have the
> >>>>>>> mechanism in
> >>>>>>>>>>> place
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> both,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this later would be a lot of
> work.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If a hypervisor doesn't support
> >>>>>>> setting
> >>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> read
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write you can always sum both up
> >>>>>>> and set
> >>>>>>>>>>> that
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> total
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limit.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why you
> >> implemented
> >>>>>>> it
> >>>>>>>>> this
> >>>>>>>>>>>>>> way?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The feature includes:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the maximum rate of VMs
> >> (in
> >>>>>>>>>>>>>>>>> disk_offering,
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) change the maximum rate of
> VMs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) limit the disk rate (total
> bps
> >>>>>>> and
> >>>>>>>>> iops)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JIRA ticket:
> >>>>>>>>> https://issues.apache.org/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> jira/browse/CLOUDSTACK-1192<ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1192<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FS (I will update later) :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******
> >>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://
> >> ****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://cwiki.apache.org/confluence/**>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<
> >>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Merge check list :-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Did you check the branch's RAT
> >>>>>>> execution
> >>>>>>>>>>>>>>>>> success?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Are there new dependencies
> >>>>>>> introduced?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What automated testing (unit
> and
> >>>>>>>>>>>>>> integration)
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> included
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Unit tests are added.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What testing has been done to
> >>>>>>> check for
> >>>>>>>>>>>>>>>>> potential
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regressions?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the bytes rate and IOPS
> >>>>>>> rate on
> >>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>>>>> UI.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) VM operations, including
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deploy, stop, start, reboot,
> >>>>>>> destroy,
> >>>>>>>>>>> expunge.
> >>>>>>>>>>>>>>>>>>>>>> migrate,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) Volume operations, including
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Attach, Detach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To review the code, you can try
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> git diff
> >>>>>>>>>>>>>>>>>>>>>> c30057635d04a2396f84c588127d7e******be42e503a7
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> f2e5591b710d04cc86815044f5823e******73a4a58944
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******
> >>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling<https://
> >> ****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/****
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://cwiki.apache.org/confluence/**>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<
> >>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>
> >> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] refs/heads/disk_io_throttling
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> issues.apache.org/****jira/browse/CLOUDSTACK-1301<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1301<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <ht**tps://
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> issues.apache.org/****jira/**browse/CLOUDSTACK-2071
> >>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071
> >>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <**
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071
> >>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <h**ttps://
> >>>>>>>>>>>>>>>>>>>>>>>>> issues.apache.org/jira/**browse/CLOUDSTACK-2071<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (**CLOUDSTACK-1301
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -     VM Disk I/O Throttling)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer,
> >>>>>>> SolidFire
> >>>>>>>>> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses
> >> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer,
> SolidFire
> >>>>>>> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
> >>>>>>> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
> >>>>>>> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >> cloud<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://solidfire.com/solution/overview/?video=play
> >>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
> Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>>>> http://solidfire.com/solution/overview/?video=play
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>> http://solidfire.com/solution/overview/?video=play
> >>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>>>>> cloud<
> >>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>> cloud<
> http://solidfire.com/solution/overview/?video=play
> >>>
> >>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>> *™*
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>> o: 303.746.7302
> >>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>> *™*
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *Mike Tutkowski*
> >>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>> o: 303.746.7302
> >>>>>>>> Advancing the way the world uses the
> >>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>> *™*
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud<
> >> http://solidfire.com/solution/overview/?video=play>
> >>>>>> *™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the cloud<
> >> http://solidfire.com/solution/overview/?video=play>
> >>>>> *™*
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Mike Tutkowski*
> >>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>> e: mike.tutkowski@solidfire.com
> >>>> o: 303.746.7302
> >>>> Advancing the way the world uses the cloud<
> >> http://solidfire.com/solution/overview/?video=play>
> >>>> *™*
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the cloud<
> >> http://solidfire.com/solution/overview/?video=play>
> >>> *™*
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> *Mike Tutkowski*
> >> *Senior CloudStack Developer, SolidFire Inc.*
> >> e: mike.tutkowski@solidfire.com
> >> o: 303.746.7302
> >> Advancing the way the world uses the
> >> cloud<http://solidfire.com/solution/overview/?video=play>
> >> *™*
> >>
>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message