cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Burwell <jburw...@basho.com>
Subject Re: [MERGE] disk_io_throttling to MASTER (Second Round)
Date Thu, 13 Jun 2013 14:34:10 GMT
Wei,

There are open questions on the thread regarding mutual exclusion of
hypervisor throttled I/O and storage provisioned IOPS.  We need to
understand how and where it will be implemented in both the UI and
service layers.  Also, can you resend the Review Board review?  My
email search skills have failed to find it.

Thanks,
-John


On Jun 13, 2013, at 3:15 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:

> If nobody object, I will merge into master today.
>
> -Wei
>
>
> 2013/6/11 John Burwell <jburwell@basho.com>
>
>> Mike,
>>
>> From a CloudStack perspective, it will keep implementation specific
>> concepts from the base data model, and provide a great test case to develop
>> a mechanism to capture this information in 4.3.  Ideally, I want CloudStack
>> to exploit these implementation specific features.  I just want to provide
>> a facility to manage that data that does not leak across implementations.
>>
>> Thanks,
>> -John
>>
>> On Jun 10, 2013, at 6:07 PM, Mike Tutkowski <mike.tutkowski@solidfire.com>
>> wrote:
>>
>>> I am pretty sure it is not a blocker for me to drop Burst IOPS. Let me
>> run
>>> that past Product Management, but I don't think it's a problem.
>>>
>>>
>>> On Mon, Jun 10, 2013 at 4:01 PM, John Burwell <jburwell@basho.com>
>> wrote:
>>>
>>>> Mike,
>>>>
>>>> Yes.  I realize my other reply did not explicitly state leaving the Min
>>>> and Max IOPS fields in the data model as these seem to generic terms
>> across
>>>> storage implementations.
>>>>
>>>> Thanks,
>>>> -John
>>>>
>>>> On Jun 10, 2013, at 5:57 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com>
>>>> wrote:
>>>>
>>>>> More generally speaking, you're looking to remove Burst IOPS from
>>>>> CloudStack for 4.2, but we would keep Min and Max (and they would be
>>>>> displayed in the Disk Offering dialog as I've proposed)?
>>>>>
>>>>>
>>>>> On Mon, Jun 10, 2013 at 3:52 PM, Mike Tutkowski <
>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>>> Just to make sure I understand your request, are you looking to
>> display
>>>>>> Min and Max (as long as Wei's feature is not in use), but not display
>>>> Burst
>>>>>> IOPS?
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 10, 2013 at 3:50 PM, John Burwell <jburwell@basho.com>
>>>> wrote:
>>>>>>
>>>>>>> Mike,
>>>>>>>
>>>>>>> My concern becomes that we start ballooning the data model and
user
>>>>>>> interface with a fields that are documented as, "If using SolidFire,
>>>> then
>>>>>>> Burst IOPS is honored and foo and bar are not.  For solution
X, Burst
>>>> IOPS
>>>>>>> is ignored, but foo and bar apply."  It may have to hold until
4.3,
>>>> but it
>>>>>>> seems like we need an extended data concept for storage drivers
that
>>>> allow
>>>>>>> them to define an additional set of properties, and persist them
into
>>>> the
>>>>>>> database as a JSON document.  Such an enhancement would also
require
>>>> some
>>>>>>> UI fanciness to consume the metadata provided by the driver and
>> adjust
>>>> the
>>>>>>> UI.  Would it be possible to defer Burst IOPS until 4.3 when
we could
>>>>>>> address extended driver data in a holistic manner?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -John
>>>>>>>
>>>>>>> On Jun 10, 2013, at 4:44 PM, Mike Tutkowski <
>>>> mike.tutkowski@solidfire.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> My thinking is that Min and Max are industry standard and
Burst is a
>>>> new
>>>>>>>> concept that could catch on.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 10, 2013 at 2:29 PM, John Burwell <jburwell@basho.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Wei,
>>>>>>>>>
>>>>>>>>> In this case, we can have the hypervisor or storage providing
the
>>>>>>> quality
>>>>>>>>> of service guarantee.  Naively, it seems reasonable to
separate
>>>>>>> hypervisor
>>>>>>>>> and storage QoS parameters and enforce mutually exclusion.
 Not
>> only
>>>>>>> does
>>>>>>>>> this approach simplify a whole range of operations, it
also avoids
>>>>>>>>> reconciling the data models.  The only question I have
about the
>>>>>>> storage
>>>>>>>>> provision IOPS is whether or not "Burst IOPS" is an industry
>> standard
>>>>>>> or
>>>>>>>>> vendor specific concept.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Jun 10, 2013, at 3:59 PM, Wei ZHOU <ustcweizhou@gmail.com>
>> wrote:
>>>>>>>>>
>>>>>>>>>> Mike,
>>>>>>>>>> I do not think users can select only one of them,
as they are
>>>>>>> implemented
>>>>>>>>>> on different sides.
>>>>>>>>>> Have you investigated the parameters other storage
devices
>> support,
>>>>>>>>> besides
>>>>>>>>>> min/max/burst IOPS? You'd better add all possible
fields in your
>>>>>>>>>> implementation.
>>>>>>>>>>
>>>>>>>>>> What do you think about this?
>>>>>>>>>> Hypersivor IOPS is fixed, and there is a drop-down
box which
>>>> includes
>>>>>>> all
>>>>>>>>>> supported storage vendors.
>>>>>>>>>> If users select "SolidFire", min/max/burst IOPS will
appear.
>>>>>>>>>> If users select other vendors, relevant fields will
appear.
>>>>>>>>>> Actually I still insist that it is better to add
the
>> storage-related
>>>>>>>>> fields
>>>>>>>>>> in another table.
>>>>>>>>>>
>>>>>>>>>> -Wei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/6/10 Mike Tutkowski <mike.tutkowski@solidfire.com>
>>>>>>>>>>
>>>>>>>>>>> Here is my thinking:
>>>>>>>>>>>
>>>>>>>>>>> Two radio buttons (whatever we want to call them):
>>>>>>>>>>>
>>>>>>>>>>> 1) Hypervisor IOPS
>>>>>>>>>>> 2) Storage IOPS
>>>>>>>>>>>
>>>>>>>>>>> Leave them both un-checked by default.
>>>>>>>>>>>
>>>>>>>>>>> If the user checks one or the other, the relevant
fields appear.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 10, 2013 at 1:38 PM, Mike Tutkowski
<
>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What do you think, Wei?
>>>>>>>>>>>>
>>>>>>>>>>>> Should we come up with a way for only one
feature (yours or
>> mine)
>>>>>>> to be
>>>>>>>>>>>> used at a time on the new Disk Offering dialog?
>>>>>>>>>>>>
>>>>>>>>>>>> Since most storage-side provisioned IOPS
don't break it down
>> into
>>>>>>>>>>> separate
>>>>>>>>>>>> read and write categories, I think that's
the way to go (only
>> one
>>>>>>>>> feature
>>>>>>>>>>>> or the other).
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions from a usability standpoint
how we want to
>>>> implement
>>>>>>>>>>> this?
>>>>>>>>>>>> It could be as simple as a radio button to
turn on your feature
>>>> and
>>>>>>>>> mine
>>>>>>>>>>>> off or vice versa.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 10, 2013 at 1:33 PM, John Burwell
<
>> jburwell@basho.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Mike,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree -- I can't image a situation
where you would want to
>> use
>>>>>>> IOPS
>>>>>>>>>>>>> provisioned by both the hypervisor and
storage.  There are two
>>>>>>> points
>>>>>>>>> of
>>>>>>>>>>>>> concern -- the UI and the management
server.  We have to ensure
>>>>>>> that
>>>>>>>>> the
>>>>>>>>>>>>> user can't create a VM from a compute/disk
offering combination
>>>>>>> where
>>>>>>>>>>>>> hypervisor throttled I/O would contradict/conflict
with storage
>>>>>>>>>>> provisioned
>>>>>>>>>>>>> IOPS.  I think this functional conflict
must be resolved in the
>>>>>>>>>>> management
>>>>>>>>>>>>> server to ensure that API calls are properly
validated with a
>> UX
>>>>>>> that
>>>>>>>>>>>>> avoids user confusion.  Have Wei and
you worked out an approach
>>>> to
>>>>>>>>>>>>> resolving this conflict?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jun 10, 2013, at 3:24 PM, Mike Tutkowski
<
>>>>>>>>>>> mike.tutkowski@solidfire.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Wei has sent me the screen shots.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't support Compute Offerings
for 4.2, so that's not an
>>>> issue
>>>>>>>>>>> here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do support Disk Offerings.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It looks like Wei has added four
new fields to the Disk
>>>> Offering.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have added three (Min, Max, and
Burst IOPS).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We just need to decide if we should
toggle between his and
>> mine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I doubt a user would want to use
both features at the same
>> time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 12:30 PM,
John Burwell <
>>>>>>> jburwell@basho.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Have Wei and you figured out
the system level as well (e.g.
>>>>>>> allowing
>>>>>>>>>>>>>>> either storage provisioned IOPS
or hypervisor throttling, but
>>>> no
>>>>>>>>>>> both)?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jun 10, 2013, at 2:12 PM,
Mike Tutkowski <
>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Perhaps Wei could send me
some screen shots of what he's
>>>>>>> changed in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> GUI
>>>>>>>>>>>>>>>> for his feature?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 11:56
AM, John Burwell <
>>>>>>> jburwell@basho.com
>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Wei,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have Mike Tutkowski and
you reconciled the potential
>> conflict
>>>>>>>>>>>>> between a
>>>>>>>>>>>>>>>>> throttled I/O VM and
a provisioned IOPs volume?  If so,
>> what
>>>>>>>>>>> solution
>>>>>>>>>>>>>>> did
>>>>>>>>>>>>>>>>> you select?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Jun 10, 2013, at 1:54
PM, Wei ZHOU <
>> ustcweizhou@gmail.com
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would like to merge
disk_io_throttling branch into
>> master.
>>>>>>>>>>>>>>>>>> Please review the
code on
>>>> https://reviews.apache.org/r/11782
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If nobody object,
I will merge into master in 72 hours.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Wei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/5/30 Wei ZHOU
<ustcweizhou@gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>> I would like
to merge disk_io_throttling branch into
>>>> master.
>>>>>>>>>>>>>>>>>>> If nobody object,
I will merge into master in 48 hours.
>>>>>>>>>>>>>>>>>>> The purpose is
:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Virtual machines
are running on the same storage device
>>>>>>> (local
>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> share strage).
Because of the rate limitation of device
>>>>>>> (such as
>>>>>>>>>>>>>>> iops),
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> one VM has large
disk operation, it may affect the disk
>>>>>>>>>>>>> performance of
>>>>>>>>>>>>>>>>>>> other VMs running
on the same storage device.
>>>>>>>>>>>>>>>>>>> It is neccesary
to set the maximum rate and limit the
>> disk
>>>>>>> I/O
>>>>>>>>> of
>>>>>>>>>>>>> VMs.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The feature includes:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> (1) set the maximum
rate of VMs (in disk_offering, and
>>>> global
>>>>>>>>>>>>>>>>>>> configuration)
>>>>>>>>>>>>>>>>>>> (2) change the
maximum rate of VMs
>>>>>>>>>>>>>>>>>>> (3) limit the
disk rate (total bps and iops)
>>>>>>>>>>>>>>>>>>> JIRA ticket:
>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1192
>>>>>>>>>>>>>>>>>>> FS (I will update
later) :
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>>>>>>>>>>>>>>>>>> Merge check list
:-
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Did you check
the branch's RAT execution success?
>>>>>>>>>>>>>>>>>>> Yes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * Are there new
dependencies introduced?
>>>>>>>>>>>>>>>>>>> No
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * What automated
testing (unit and integration) is
>> included
>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> feature?
>>>>>>>>>>>>>>>>>>> Unit tests are
added.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * What testing
has been done to check for potential
>>>>>>> regressions?
>>>>>>>>>>>>>>>>>>> (1) set the bytes
rate and IOPS rate on CloudStack UI.
>>>>>>>>>>>>>>>>>>> (2) VM operations,
including
>>>>>>>>>>>>>>>>>>> deploy, stop,
start, reboot, destroy, expunge. migrate,
>>>>>>> restore
>>>>>>>>>>>>>>>>>>> (3) Volume operations,
including
>>>>>>>>>>>>>>>>>>> Attach, Detach
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To review the
code, you can try
>>>>>>>>>>>>>>>>>>> git diff c30057635d04a2396f84c588127d7ebe42e503a7
>>>>>>>>>>>>>>>>>>> f2e5591b710d04cc86815044f5823e73a4a58944
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> Wei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>>>>>>>>>>>>>>>>>> [2] refs/heads/disk_io_throttling
>>>>>>>>>>>>>>>>>>> [3]
>> https://issues.apache.org/jira/browse/CLOUDSTACK-1301<
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071
>>>>>>>>>>>>>> (CLOUDSTACK-1301
>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>> VM Disk I/O Throttling)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>>>>>> *Senior CloudStack Developer,
SolidFire Inc.*
>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>>>>>> Advancing the way the world
uses the
>>>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>>>>>>> *™*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire
Inc.*
>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>>>> Advancing the way the world uses
the
>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>>>>> *™*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>> Advancing the way the world uses the cloud<
>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>>> *™*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>> Advancing the way the world uses the
>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>>>>> *™*
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the
>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>>>>> *™*
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Mike Tutkowski*
>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>> e: mike.tutkowski@solidfire.com
>>>>>> o: 303.746.7302
>>>>>> Advancing the way the world uses the cloud<
>>>> http://solidfire.com/solution/overview/?video=play>
>>>>>> *™*
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the
>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>>> *™*
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the
>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>
>>

Mime
View raw message