cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei ZHOU <ustcweiz...@gmail.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Mon, 03 Jun 2013 14:55:03 GMT
John,

For the billing, as no one works on billing now, users need to calculate
the billing by themselves. They can get the service_offering and
disk_offering of a VMs and volumes for calculation. Of course it is better
to tell user the exact limitation value of individual volume, and network
rate limitation for nics as well. I can work on it later. Do you think it
is a part of I/O throttling?

Sorry my misunstand the second the question.

Agree with what you said about the two features.

-Wei


2013/6/3 John Burwell <jburwell@basho.com>

> Wei,
>
>
> On Jun 3, 2013, at 2:13 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
>
> > Hi John, Mike
> >
> > I hope Mike's aswer helps you. I am trying to adding more.
> >
> > (1) I think billing should depend on IO statistics rather than IOPS
> > limitation. Please review disk_io_stat if you have time.   disk_io_stat
> can
> > get the IO statistics including bytes/iops read/write for an individual
> > virtual machine.
>
> Going by the AWS model, customers are billed more for volumes with
> provisioned IOPS, as well as, for those operations (
> http://aws.amazon.com/ebs/).  I would imagine our users would like the
> option to employ similar cost models.  Could an operator implement such a
> billing model in the current patch?
>
> >
> > (2) Do you mean IOPS runtime change? KVM supports setting IOPS/BPS
> > limitation for a running virtual machine through command line. However,
> > CloudStack does not support changing the parameters of a created offering
> > (computer offering or disk offering).
>
> I meant at the Java interface level.  I apologize for being unclear.  Can
> we more generalize allocation algorithms with a set of interfaces that
> describe the service guarantees provided by a resource?
>
> >
> > (3) It is a good question. Maybe it is better to commit Mike's patch
> after
> > disk_io_throttling as Mike needs to consider the limitation in hypervisor
> > type, I think.
>
> I will expand on my thoughts in a later response to Mike regarding the
> touch points between these two features.  I think that disk_io_throttling
> will need to be merged before SolidFire, but I think we need closer
> coordination between the branches (possibly have have solidfire track
> disk_io_throttling) to coordinate on this issue.
>
> >
> > - Wei
> >
> >
> > 2013/6/3 John Burwell <jburwell@basho.com>
> >
> >> Mike,
> >>
> >> The things I want to understand are the following:
> >>
> >>   1) Is there value in capturing IOPS policies be captured in a common
> >> data model (e.g. for billing/usage purposes, expressing offerings).
> >>    2) Should there be a common interface model for reasoning about IOP
> >> provisioning at runtime?
> >>    3) How are conflicting provisioned IOPS configurations between a
> >> hypervisor and storage device reconciled?  In particular, a scenario
> where
> >> is lead to believe (and billed) for more IOPS configured for a VM than a
> >> storage device has been configured to deliver.  Another scenario could a
> >> consistent configuration between a VM and a storage device at creation
> >> time, but a later modification to storage device introduces logical
> >> inconsistency.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> >> wrote:
> >>
> >> Hi John,
> >>
> >> I believe Wei's feature deals with controlling the max number of IOPS
> from
> >> the hypervisor side.
> >>
> >> My feature is focused on controlling IOPS from the storage system side.
> >>
> >> I hope that helps. :)
> >>
> >>
> >> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell <jburwell@basho.com>
> wrote:
> >>
> >>> Wei,
> >>>
> >>> My opinion is that no features should be merged until all functional
> >>> issues have been resolved and it is ready to turn over to test.  Until
> >> the
> >>> total Ops vs discrete read/write ops issue is addressed and re-reviewed
> >> by
> >>> Wido, I don't think this criteria has been satisfied.
> >>>
> >>> Also, how does this work intersect/compliment the SolidFire patch (
> >>> https://reviews.apache.org/r/11479/)?  As I understand it that work is
> >>> also involves provisioned IOPS.  I would like to ensure we don't have a
> >>> scenario where provisioned IOPS in KVM and SolidFire are unnecessarily
> >>> incompatible.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
> >>>
> >>> Wido,
> >>>
> >>>
> >>> Sure. I will change it next week.
> >>>
> >>>
> >>> -Wei
> >>>
> >>>
> >>>
> >>> 2013/6/1 Wido den Hollander <wido@widodh.nl>
> >>>
> >>>
> >>> Hi Wei,
> >>>
> >>>
> >>>
> >>> On 06/01/2013 08:24 AM, Wei ZHOU wrote:
> >>>
> >>>
> >>> Wido,
> >>>
> >>>
> >>> Exactly. I have pushed the features into master.
> >>>
> >>>
> >>> If anyone object thems for technical reason till Monday, I will revert
> >>>
> >>> them.
> >>>
> >>>
> >>> For the sake of clarity I just want to mention again that we should
> >> change
> >>>
> >>> the total IOps to R/W IOps asap so that we never release a version with
> >>>
> >>> only total IOps.
> >>>
> >>>
> >>> You laid the groundwork for the I/O throttling and that's great! We
> >> should
> >>>
> >>> however prevent that we create legacy from day #1.
> >>>
> >>>
> >>> Wido
> >>>
> >>>
> >>> -Wei
> >>>
> >>>
> >>>
> >>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
> >>>
> >>>
> >>> On 05/31/2013 03:59 PM, John Burwell wrote:
> >>>
> >>>
> >>> Wido,
> >>>
> >>>
> >>> +1 -- this enhancement must to discretely support read and write IOPS.
> >>>
> >>> I
> >>>
> >>> don't see how it could be fixed later because I don't see how we
> >>>
> >>> correctly
> >>>
> >>> split total IOPS into read and write.  Therefore, we would be stuck
> >>>
> >>> with a
> >>>
> >>> total unless/until we decided to break backwards compatibility.
> >>>
> >>>
> >>>
> >>> What Wei meant was merging it into master now so that it will go in the
> >>>
> >>> 4.2 branch and add Read / Write IOps before the 4.2 release so that 4.2
> >>>
> >>> will be released with Read and Write instead of Total IOps.
> >>>
> >>>
> >>> This is to make the May 31st feature freeze date. But if the window
> moves
> >>>
> >>> (see other threads) then it won't be necessary to do that.
> >>>
> >>>
> >>> Wido
> >>>
> >>>
> >>>
> >>> I also completely agree that there is no association between network
> >>>
> >>> and
> >>>
> >>>
> >>> disk I/O.
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> -John
> >>>
> >>>
> >>> On May 31, 2013, at 9:51 AM, Wido den Hollander <wido@widodh.nl>
> wrote:
> >>>
> >>>
> >>> Hi Wei,
> >>>
> >>>
> >>>
> >>> On 05/31/2013 03:13 PM, Wei ZHOU wrote:
> >>>
> >>>
> >>> Hi Wido,
> >>>
> >>>
> >>> Thanks. Good question.
> >>>
> >>>
> >>> I  thought about at the beginning. Finally I decided to ignore the
> >>>
> >>> difference of read and write mainly because the network throttling did
> >>>
> >>> not
> >>>
> >>> care the difference of sent and received bytes as well.
> >>>
> >>> That reasoning seems odd. Networking and disk I/O completely different.
> >>>
> >>>
> >>> Disk I/O is much more expensive in most situations then network
> >>>
> >>> bandwith.
> >>>
> >>>
> >>> Implementing it will be some copy-paste work. It could be
> >>>
> >>> implemented in
> >>>
> >>>
> >>> few days. For the deadline of feature freeze, I will implement it
> >>>
> >>> after
> >>>
> >>> that , if needed.
> >>>
> >>>
> >>>
> >>> It think it's a feature we can't miss. But if it goes into the 4.2
> >>>
> >>> window we have to make sure we don't release with only total IOps and
> >>>
> >>> fix
> >>>
> >>> it in 4.3, that will confuse users.
> >>>
> >>>
> >>> Wido
> >>>
> >>>
> >>> -Wei
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
> >>>
> >>>
> >>> Hi Wei,
> >>>
> >>>
> >>>
> >>>
> >>> On 05/30/2013 06:03 PM, Wei ZHOU wrote:
> >>>
> >>>
> >>> Hi,
> >>>
> >>>
> >>> I would like to merge disk_io_throttling branch into master.
> >>>
> >>> If nobody object, I will merge into master in 48 hours.
> >>>
> >>> The purpose is :
> >>>
> >>>
> >>> Virtual machines are running on the same storage device (local
> >>>
> >>> storage or
> >>>
> >>> share strage). Because of the rate limitation of device (such as
> >>>
> >>> iops), if
> >>>
> >>> one VM has large disk operation, it may affect the disk performance
> >>>
> >>> of
> >>>
> >>> other VMs running on the same storage device.
> >>>
> >>> It is neccesary to set the maximum rate and limit the disk I/O of
> >>>
> >>> VMs.
> >>>
> >>>
> >>>
> >>> Looking at the code I see you make no difference between Read and
> >>>
> >>> Write
> >>>
> >>> IOps.
> >>>
> >>>
> >>> Qemu and libvirt support setting both a different rate for Read and
> >>>
> >>> Write
> >>>
> >>> IOps which could benefit a lot of users.
> >>>
> >>>
> >>> It's also strange, in the polling side you collect both the Read and
> >>>
> >>> Write
> >>>
> >>> IOps, but on the throttling side you only go for a global value.
> >>>
> >>>
> >>> Write IOps are usually much more expensive then Read IOps, so it
> >>>
> >>> seems
> >>>
> >>> like a valid use-case where that an admin would set a lower value for
> >>>
> >>> write
> >>>
> >>> IOps vs Read IOps.
> >>>
> >>>
> >>> Since this only supports KVM at this point I think it would be of
> >>>
> >>> great
> >>>
> >>> value to at least have the mechanism in place to support both,
> >>>
> >>> implementing
> >>>
> >>> this later would be a lot of work.
> >>>
> >>>
> >>> If a hypervisor doesn't support setting different values for read and
> >>>
> >>> write you can always sum both up and set that as the total limit.
> >>>
> >>>
> >>> Can you explain why you implemented it this way?
> >>>
> >>>
> >>> Wido
> >>>
> >>>
> >>>  The feature includes:
> >>>
> >>>
> >>>
> >>> (1) set the maximum rate of VMs (in disk_offering, and global
> >>>
> >>> configuration)
> >>>
> >>> (2) change the maximum rate of VMs
> >>>
> >>> (3) limit the disk rate (total bps and iops)
> >>>
> >>> JIRA ticket: https://issues.apache.org/****
> >>>
> >>> jira/browse/CLOUDSTACK-1192<ht**tps://issues.apache.org/****
> >>>
> >>> jira/browse/CLOUDSTACK-1192<
> >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
> >>>
> >>> <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1192<
> >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192>
> >>>
> >>> <**
> >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
> >>>
> >>>
> >>>
> >>> FS (I will update later) :
> >>>
> >>>
> >>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
> >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> >>>
> >>> <
> >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<
> >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
> >>>
> >>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****<
> >>> http://cwiki.apache.org/confluence/**>
> >>>
> >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
> >>>
> >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling<
> >>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>
> >>>
> >>>
> >>>
> >>> Merge check list :-
> >>>
> >>>
> >>> * Did you check the branch's RAT execution success?
> >>>
> >>> Yes
> >>>
> >>>
> >>> * Are there new dependencies introduced?
> >>>
> >>> No
> >>>
> >>>
> >>> * What automated testing (unit and integration) is included in the
> >>>
> >>> new
> >>>
> >>> feature?
> >>>
> >>> Unit tests are added.
> >>>
> >>>
> >>> * What testing has been done to check for potential regressions?
> >>>
> >>> (1) set the bytes rate and IOPS rate on CloudStack UI.
> >>>
> >>> (2) VM operations, including
> >>>
> >>> deploy, stop, start, reboot, destroy, expunge. migrate, restore
> >>>
> >>> (3) Volume operations, including
> >>>
> >>> Attach, Detach
> >>>
> >>>
> >>> To review the code, you can try
> >>>
> >>>  git diff c30057635d04a2396f84c588127d7e******be42e503a7
> >>>
> >>> f2e5591b710d04cc86815044f5823e******73a4a58944
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Wei
> >>>
> >>>
> >>> [1]
> >>>
> >>>
> >>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
> >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> >>>
> >>> <
> >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<
> >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
> >>>
> >>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****<
> >>> http://cwiki.apache.org/confluence/**>
> >>>
> >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
> >>>
> >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling<
> >>>
> >>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> >>>>
> >>>
> >>>
> >>>
> >>> [2] refs/heads/disk_io_throttling
> >>>
> >>> [3]
> >>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301<
> >> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301>
> >>>
> >>> <ht**tps://issues.apache.org/****jira/browse/CLOUDSTACK-1301<
> >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
> >>>
> >>> <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1301<
> >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301>
> >>>
> >>> <**
> >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
> >>>
> >>>
> >>>
> >>> <ht**tps://issues.apache.org/****jira/**browse/CLOUDSTACK-2071<
> >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071>
> >>>
> >>> **<
> >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071<
> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071>
> >>>
> >>> <**
> >>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071<
> >> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
> >>>
> >>> <h**ttps://issues.apache.org/jira/**browse/CLOUDSTACK-2071<
> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
> >>>
> >>>
> >>>
> >>> (**CLOUDSTACK-1301
> >>>
> >>> -     VM Disk I/O Throttling)
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> *Mike Tutkowski*
> >> *Senior CloudStack Developer, SolidFire Inc.*
> >> e: mike.tutkowski@solidfire.com
> >> o: 303.746.7302
> >> Advancing the way the world uses the
> >> cloud<http://solidfire.com/solution/overview/?video=play>
> >> *™*
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message