cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Mon, 03 Jun 2013 01:14:58 GMT
Sending to the CS e-mail list (I accidentally only sent this to John
before).


On Sun, Jun 2, 2013 at 7:10 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Hi John,
>
> Let me see if I can answer your questions.
>
> 1) If we mean provide the ability to charge more for higher IOPS Disk
> Offerings, then I would say 'yes.'
>
> 2) I might need more detail on what you're thinking here.
>
> 3) The IOPS in my patch code are from the storage system's point of view
> only. If the hypervisor has a Max that is, say, below the Max of the
> storage system, the storage system will never hit its Max. That being the
> case, it probably makes sense to provide an error message to the user in
> such a situation. I haven't actually seen Wei's code yet, but we should
> coordinate on activities like this.
>
>
> On Sun, Jun 2, 2013 at 6:54 PM, John Burwell <jburwell@basho.com> wrote:
>
>> Mike,
>>
>> The things I want to understand are the following:
>>
>>    1) Is there value in capturing IOPS policies be captured in a common
>> data model (e.g. for billing/usage purposes, expressing offerings).
>>     2) Should there be a common interface model for reasoning about IOP
>> provisioning at runtime?
>>     3) How are conflicting provisioned IOPS configurations between a
>> hypervisor and storage device reconciled?  In particular, a scenario where
>> is lead to believe (and billed) for more IOPS configured for a VM than a
>> storage device has been configured to deliver.  Another scenario could a
>> consistent configuration between a VM and a storage device at creation
>> time, but a later modification to storage device introduces logical
>> inconsistency.
>>
>> Thanks,
>> -John
>>
>> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski <mike.tutkowski@solidfire.com>
>> wrote:
>>
>>  Hi John,
>>
>> I believe Wei's feature deals with controlling the max number of IOPS
>> from the hypervisor side.
>>
>> My feature is focused on controlling IOPS from the storage system side.
>>
>> I hope that helps. :)
>>
>>
>> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell <jburwell@basho.com> wrote:
>>
>>> Wei,
>>>
>>> My opinion is that no features should be merged until all functional
>>> issues have been resolved and it is ready to turn over to test.  Until the
>>> total Ops vs discrete read/write ops issue is addressed and re-reviewed by
>>> Wido, I don't think this criteria has been satisfied.
>>>
>>> Also, how does this work intersect/compliment the SolidFire patch (
>>> https://reviews.apache.org/r/11479/)?  As I understand it that work is
>>> also involves provisioned IOPS.  I would like to ensure we don't have a
>>> scenario where provisioned IOPS in KVM and SolidFire are unnecessarily
>>> incompatible.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
>>>
>>> Wido,
>>>
>>>
>>> Sure. I will change it next week.
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>> 2013/6/1 Wido den Hollander <wido@widodh.nl>
>>>
>>>
>>> Hi Wei,
>>>
>>>
>>>
>>> On 06/01/2013 08:24 AM, Wei ZHOU wrote:
>>>
>>>
>>> Wido,
>>>
>>>
>>> Exactly. I have pushed the features into master.
>>>
>>>
>>> If anyone object thems for technical reason till Monday, I will revert
>>>
>>> them.
>>>
>>>
>>> For the sake of clarity I just want to mention again that we should
>>> change
>>>
>>> the total IOps to R/W IOps asap so that we never release a version with
>>>
>>> only total IOps.
>>>
>>>
>>> You laid the groundwork for the I/O throttling and that's great! We
>>> should
>>>
>>> however prevent that we create legacy from day #1.
>>>
>>>
>>> Wido
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
>>>
>>>
>>> On 05/31/2013 03:59 PM, John Burwell wrote:
>>>
>>>
>>> Wido,
>>>
>>>
>>> +1 -- this enhancement must to discretely support read and write IOPS.
>>>
>>> I
>>>
>>> don't see how it could be fixed later because I don't see how we
>>>
>>> correctly
>>>
>>> split total IOPS into read and write.  Therefore, we would be stuck
>>>
>>> with a
>>>
>>> total unless/until we decided to break backwards compatibility.
>>>
>>>
>>>
>>> What Wei meant was merging it into master now so that it will go in the
>>>
>>> 4.2 branch and add Read / Write IOps before the 4.2 release so that 4.2
>>>
>>> will be released with Read and Write instead of Total IOps.
>>>
>>>
>>> This is to make the May 31st feature freeze date. But if the window moves
>>>
>>> (see other threads) then it won't be necessary to do that.
>>>
>>>
>>> Wido
>>>
>>>
>>>
>>>  I also completely agree that there is no association between network
>>>
>>> and
>>>
>>>
>>> disk I/O.
>>>
>>>
>>> Thanks,
>>>
>>> -John
>>>
>>>
>>> On May 31, 2013, at 9:51 AM, Wido den Hollander <wido@widodh.nl> wrote:
>>>
>>>
>>>  Hi Wei,
>>>
>>>
>>>
>>> On 05/31/2013 03:13 PM, Wei ZHOU wrote:
>>>
>>>
>>> Hi Wido,
>>>
>>>
>>> Thanks. Good question.
>>>
>>>
>>> I  thought about at the beginning. Finally I decided to ignore the
>>>
>>> difference of read and write mainly because the network throttling did
>>>
>>> not
>>>
>>> care the difference of sent and received bytes as well.
>>>
>>> That reasoning seems odd. Networking and disk I/O completely different.
>>>
>>>
>>> Disk I/O is much more expensive in most situations then network
>>>
>>> bandwith.
>>>
>>>
>>>  Implementing it will be some copy-paste work. It could be
>>>
>>> implemented in
>>>
>>>
>>> few days. For the deadline of feature freeze, I will implement it
>>>
>>> after
>>>
>>>  that , if needed.
>>>
>>>
>>>
>>> It think it's a feature we can't miss. But if it goes into the 4.2
>>>
>>>  window we have to make sure we don't release with only total IOps and
>>>
>>> fix
>>>
>>> it in 4.3, that will confuse users.
>>>
>>>
>>> Wido
>>>
>>>
>>>  -Wei
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
>>>
>>>
>>>  Hi Wei,
>>>
>>>
>>>
>>>
>>> On 05/30/2013 06:03 PM, Wei ZHOU wrote:
>>>
>>>
>>>  Hi,
>>>
>>>
>>>  I would like to merge disk_io_throttling branch into master.
>>>
>>>  If nobody object, I will merge into master in 48 hours.
>>>
>>> The purpose is :
>>>
>>>
>>> Virtual machines are running on the same storage device (local
>>>
>>> storage or
>>>
>>>  share strage). Because of the rate limitation of device (such as
>>>
>>> iops), if
>>>
>>>  one VM has large disk operation, it may affect the disk performance
>>>
>>>  of
>>>
>>> other VMs running on the same storage device.
>>>
>>> It is neccesary to set the maximum rate and limit the disk I/O of
>>>
>>>  VMs.
>>>
>>>
>>>
>>>  Looking at the code I see you make no difference between Read and
>>>
>>> Write
>>>
>>> IOps.
>>>
>>>
>>> Qemu and libvirt support setting both a different rate for Read and
>>>
>>>  Write
>>>
>>> IOps which could benefit a lot of users.
>>>
>>>
>>>  It's also strange, in the polling side you collect both the Read and
>>>
>>>  Write
>>>
>>> IOps, but on the throttling side you only go for a global value.
>>>
>>>
>>> Write IOps are usually much more expensive then Read IOps, so it
>>>
>>> seems
>>>
>>> like a valid use-case where that an admin would set a lower value for
>>>
>>>  write
>>>
>>> IOps vs Read IOps.
>>>
>>>
>>>  Since this only supports KVM at this point I think it would be of
>>>
>>> great
>>>
>>> value to at least have the mechanism in place to support both,
>>>
>>> implementing
>>>
>>> this later would be a lot of work.
>>>
>>>
>>>  If a hypervisor doesn't support setting different values for read and
>>>
>>> write you can always sum both up and set that as the total limit.
>>>
>>>
>>> Can you explain why you implemented it this way?
>>>
>>>
>>> Wido
>>>
>>>
>>>   The feature includes:
>>>
>>>
>>>
>>> (1) set the maximum rate of VMs (in disk_offering, and global
>>>
>>>  configuration)
>>>
>>>  (2) change the maximum rate of VMs
>>>
>>> (3) limit the disk rate (total bps and iops)
>>>
>>>  JIRA ticket: https://issues.apache.org/****
>>>
>>>  jira/browse/CLOUDSTACK-1192<ht**tps://issues.apache.org/****
>>>
>>>  jira/browse/CLOUDSTACK-1192<
>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
>>>
>>>  <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1192<
>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192>
>>>
>>> <**
>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
>>>
>>>
>>>
>>>  FS (I will update later) :
>>>
>>>
>>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
>>>
>>> <
>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
>>>
>>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****<
>>> http://cwiki.apache.org/confluence/**>
>>>
>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
>>>
>>>  apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling<
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>> >
>>>
>>>
>>>
>>>  Merge check list :-
>>>
>>>
>>> * Did you check the branch's RAT execution success?
>>>
>>> Yes
>>>
>>>
>>>  * Are there new dependencies introduced?
>>>
>>>  No
>>>
>>>
>>>  * What automated testing (unit and integration) is included in the
>>>
>>> new
>>>
>>> feature?
>>>
>>>  Unit tests are added.
>>>
>>>
>>>  * What testing has been done to check for potential regressions?
>>>
>>> (1) set the bytes rate and IOPS rate on CloudStack UI.
>>>
>>> (2) VM operations, including
>>>
>>>  deploy, stop, start, reboot, destroy, expunge. migrate, restore
>>>
>>> (3) Volume operations, including
>>>
>>> Attach, Detach
>>>
>>>
>>> To review the code, you can try
>>>
>>>   git diff c30057635d04a2396f84c588127d7e******be42e503a7
>>>
>>> f2e5591b710d04cc86815044f5823e******73a4a58944
>>>
>>>
>>> Best regards,
>>>
>>> Wei
>>>
>>>
>>>  [1]
>>>
>>>
>>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
>>>
>>> <
>>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
>>>
>>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****<
>>> http://cwiki.apache.org/confluence/**>
>>>
>>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
>>>
>>>  apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling<
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
>>> >
>>>
>>>
>>>
>>> [2] refs/heads/disk_io_throttling
>>>
>>>  [3]
>>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301<https://issues.apache.org/****jira/browse/CLOUDSTACK-1301>
>>>
>>> <ht**tps://issues.apache.org/****jira/browse/CLOUDSTACK-1301<
>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
>>>
>>> <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1301<
>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301>
>>>
>>> <**
>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
>>>
>>>
>>>
>>> <ht**tps://issues.apache.org/****jira/**browse/CLOUDSTACK-2071<
>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071>
>>>
>>> **<
>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071<http://issues.apache.org/jira/**browse/CLOUDSTACK-2071>
>>>
>>> <**
>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071<https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
>>>
>>> <h**ttps://issues.apache.org/jira/**browse/CLOUDSTACK-2071<
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
>>>
>>>
>>>
>>>  (**CLOUDSTACK-1301
>>>
>>>  -     VM Disk I/O Throttling)
>>>
>>>
>>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
>> *™*
>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message