cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Mon, 03 Jun 2013 17:21:36 GMT
Yes, ultimately I would like to support all hypervisors that CloudStack
supports. I think I'm just out of time for 4.2 to get KVM in.

Right now this plug-in supports XenServer. Depending on what we do with
regards to 4.2 feature freeze, I have it working for VMware in my sandbox,
as well.

Also, just to be clear, this is all in regards to Disk Offerings. I plan to
support Compute Offerings post 4.2.


On Mon, Jun 3, 2013 at 11:14 AM, Kelcey Jamison Damage <kelcey@bbits.ca>wrote:

> Is there any plan on supporting KVM in the patch cycle post 4.2?
>
> ----- Original Message -----
> From: "Mike Tutkowski" <mike.tutkowski@solidfire.com>
> To: dev@cloudstack.apache.org
> Sent: Monday, June 3, 2013 10:12:32 AM
> Subject: Re: [MERGE] disk_io_throttling to MASTER
>
> I agree on merging Wei's feature first, then mine.
>
> If his feature is for KVM only, then it is a non issue as I don't support
> KVM in 4.2.
>
>
> On Mon, Jun 3, 2013 at 8:55 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
>
> > John,
> >
> > For the billing, as no one works on billing now, users need to calculate
> > the billing by themselves. They can get the service_offering and
> > disk_offering of a VMs and volumes for calculation. Of course it is
> better
> > to tell user the exact limitation value of individual volume, and network
> > rate limitation for nics as well. I can work on it later. Do you think it
> > is a part of I/O throttling?
> >
> > Sorry my misunstand the second the question.
> >
> > Agree with what you said about the two features.
> >
> > -Wei
> >
> >
> > 2013/6/3 John Burwell <jburwell@basho.com>
> >
> > > Wei,
> > >
> > >
> > > On Jun 3, 2013, at 2:13 AM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
> > >
> > > > Hi John, Mike
> > > >
> > > > I hope Mike's aswer helps you. I am trying to adding more.
> > > >
> > > > (1) I think billing should depend on IO statistics rather than IOPS
> > > > limitation. Please review disk_io_stat if you have time.
> disk_io_stat
> > > can
> > > > get the IO statistics including bytes/iops read/write for an
> individual
> > > > virtual machine.
> > >
> > > Going by the AWS model, customers are billed more for volumes with
> > > provisioned IOPS, as well as, for those operations (
> > > http://aws.amazon.com/ebs/).  I would imagine our users would like the
> > > option to employ similar cost models.  Could an operator implement
> such a
> > > billing model in the current patch?
> > >
> > > >
> > > > (2) Do you mean IOPS runtime change? KVM supports setting IOPS/BPS
> > > > limitation for a running virtual machine through command line.
> However,
> > > > CloudStack does not support changing the parameters of a created
> > offering
> > > > (computer offering or disk offering).
> > >
> > > I meant at the Java interface level.  I apologize for being unclear.
>  Can
> > > we more generalize allocation algorithms with a set of interfaces that
> > > describe the service guarantees provided by a resource?
> > >
> > > >
> > > > (3) It is a good question. Maybe it is better to commit Mike's patch
> > > after
> > > > disk_io_throttling as Mike needs to consider the limitation in
> > hypervisor
> > > > type, I think.
> > >
> > > I will expand on my thoughts in a later response to Mike regarding the
> > > touch points between these two features.  I think that
> disk_io_throttling
> > > will need to be merged before SolidFire, but I think we need closer
> > > coordination between the branches (possibly have have solidfire track
> > > disk_io_throttling) to coordinate on this issue.
> > >
> > > >
> > > > - Wei
> > > >
> > > >
> > > > 2013/6/3 John Burwell <jburwell@basho.com>
> > > >
> > > >> Mike,
> > > >>
> > > >> The things I want to understand are the following:
> > > >>
> > > >>   1) Is there value in capturing IOPS policies be captured in a
> common
> > > >> data model (e.g. for billing/usage purposes, expressing offerings).
> > > >>    2) Should there be a common interface model for reasoning about
> IOP
> > > >> provisioning at runtime?
> > > >>    3) How are conflicting provisioned IOPS configurations between
a
> > > >> hypervisor and storage device reconciled?  In particular, a scenario
> > > where
> > > >> is lead to believe (and billed) for more IOPS configured for a VM
> > than a
> > > >> storage device has been configured to deliver.  Another scenario
> > could a
> > > >> consistent configuration between a VM and a storage device at
> creation
> > > >> time, but a later modification to storage device introduces logical
> > > >> inconsistency.
> > > >>
> > > >> Thanks,
> > > >> -John
> > > >>
> > > >> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com>
> > > >> wrote:
> > > >>
> > > >> Hi John,
> > > >>
> > > >> I believe Wei's feature deals with controlling the max number of
> IOPS
> > > from
> > > >> the hypervisor side.
> > > >>
> > > >> My feature is focused on controlling IOPS from the storage system
> > side.
> > > >>
> > > >> I hope that helps. :)
> > > >>
> > > >>
> > > >> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell <jburwell@basho.com>
> > > wrote:
> > > >>
> > > >>> Wei,
> > > >>>
> > > >>> My opinion is that no features should be merged until all
> functional
> > > >>> issues have been resolved and it is ready to turn over to test.
> >  Until
> > > >> the
> > > >>> total Ops vs discrete read/write ops issue is addressed and
> > re-reviewed
> > > >> by
> > > >>> Wido, I don't think this criteria has been satisfied.
> > > >>>
> > > >>> Also, how does this work intersect/compliment the SolidFire patch
(
> > > >>> https://reviews.apache.org/r/11479/)?  As I understand it that
> work
> > is
> > > >>> also involves provisioned IOPS.  I would like to ensure we don't
> > have a
> > > >>> scenario where provisioned IOPS in KVM and SolidFire are
> > unnecessarily
> > > >>> incompatible.
> > > >>>
> > > >>> Thanks,
> > > >>> -John
> > > >>>
> > > >>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU <ustcweizhou@gmail.com>
> wrote:
> > > >>>
> > > >>> Wido,
> > > >>>
> > > >>>
> > > >>> Sure. I will change it next week.
> > > >>>
> > > >>>
> > > >>> -Wei
> > > >>>
> > > >>>
> > > >>>
> > > >>> 2013/6/1 Wido den Hollander <wido@widodh.nl>
> > > >>>
> > > >>>
> > > >>> Hi Wei,
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 06/01/2013 08:24 AM, Wei ZHOU wrote:
> > > >>>
> > > >>>
> > > >>> Wido,
> > > >>>
> > > >>>
> > > >>> Exactly. I have pushed the features into master.
> > > >>>
> > > >>>
> > > >>> If anyone object thems for technical reason till Monday, I will
> > revert
> > > >>>
> > > >>> them.
> > > >>>
> > > >>>
> > > >>> For the sake of clarity I just want to mention again that we should
> > > >> change
> > > >>>
> > > >>> the total IOps to R/W IOps asap so that we never release a version
> > with
> > > >>>
> > > >>> only total IOps.
> > > >>>
> > > >>>
> > > >>> You laid the groundwork for the I/O throttling and that's great!
We
> > > >> should
> > > >>>
> > > >>> however prevent that we create legacy from day #1.
> > > >>>
> > > >>>
> > > >>> Wido
> > > >>>
> > > >>>
> > > >>> -Wei
> > > >>>
> > > >>>
> > > >>>
> > > >>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
> > > >>>
> > > >>>
> > > >>> On 05/31/2013 03:59 PM, John Burwell wrote:
> > > >>>
> > > >>>
> > > >>> Wido,
> > > >>>
> > > >>>
> > > >>> +1 -- this enhancement must to discretely support read and write
> > IOPS.
> > > >>>
> > > >>> I
> > > >>>
> > > >>> don't see how it could be fixed later because I don't see how
we
> > > >>>
> > > >>> correctly
> > > >>>
> > > >>> split total IOPS into read and write.  Therefore, we would be
stuck
> > > >>>
> > > >>> with a
> > > >>>
> > > >>> total unless/until we decided to break backwards compatibility.
> > > >>>
> > > >>>
> > > >>>
> > > >>> What Wei meant was merging it into master now so that it will
go in
> > the
> > > >>>
> > > >>> 4.2 branch and add Read / Write IOps before the 4.2 release so
that
> > 4.2
> > > >>>
> > > >>> will be released with Read and Write instead of Total IOps.
> > > >>>
> > > >>>
> > > >>> This is to make the May 31st feature freeze date. But if the window
> > > moves
> > > >>>
> > > >>> (see other threads) then it won't be necessary to do that.
> > > >>>
> > > >>>
> > > >>> Wido
> > > >>>
> > > >>>
> > > >>>
> > > >>> I also completely agree that there is no association between
> network
> > > >>>
> > > >>> and
> > > >>>
> > > >>>
> > > >>> disk I/O.
> > > >>>
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> -John
> > > >>>
> > > >>>
> > > >>> On May 31, 2013, at 9:51 AM, Wido den Hollander <wido@widodh.nl>
> > > wrote:
> > > >>>
> > > >>>
> > > >>> Hi Wei,
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 05/31/2013 03:13 PM, Wei ZHOU wrote:
> > > >>>
> > > >>>
> > > >>> Hi Wido,
> > > >>>
> > > >>>
> > > >>> Thanks. Good question.
> > > >>>
> > > >>>
> > > >>> I  thought about at the beginning. Finally I decided to ignore
the
> > > >>>
> > > >>> difference of read and write mainly because the network throttling
> > did
> > > >>>
> > > >>> not
> > > >>>
> > > >>> care the difference of sent and received bytes as well.
> > > >>>
> > > >>> That reasoning seems odd. Networking and disk I/O completely
> > different.
> > > >>>
> > > >>>
> > > >>> Disk I/O is much more expensive in most situations then network
> > > >>>
> > > >>> bandwith.
> > > >>>
> > > >>>
> > > >>> Implementing it will be some copy-paste work. It could be
> > > >>>
> > > >>> implemented in
> > > >>>
> > > >>>
> > > >>> few days. For the deadline of feature freeze, I will implement
it
> > > >>>
> > > >>> after
> > > >>>
> > > >>> that , if needed.
> > > >>>
> > > >>>
> > > >>>
> > > >>> It think it's a feature we can't miss. But if it goes into the
4.2
> > > >>>
> > > >>> window we have to make sure we don't release with only total IOps
> and
> > > >>>
> > > >>> fix
> > > >>>
> > > >>> it in 4.3, that will confuse users.
> > > >>>
> > > >>>
> > > >>> Wido
> > > >>>
> > > >>>
> > > >>> -Wei
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> 2013/5/31 Wido den Hollander <wido@widodh.nl>
> > > >>>
> > > >>>
> > > >>> Hi Wei,
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 05/30/2013 06:03 PM, Wei ZHOU wrote:
> > > >>>
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>>
> > > >>> I would like to merge disk_io_throttling branch into master.
> > > >>>
> > > >>> If nobody object, I will merge into master in 48 hours.
> > > >>>
> > > >>> The purpose is :
> > > >>>
> > > >>>
> > > >>> Virtual machines are running on the same storage device (local
> > > >>>
> > > >>> storage or
> > > >>>
> > > >>> share strage). Because of the rate limitation of device (such
as
> > > >>>
> > > >>> iops), if
> > > >>>
> > > >>> one VM has large disk operation, it may affect the disk performance
> > > >>>
> > > >>> of
> > > >>>
> > > >>> other VMs running on the same storage device.
> > > >>>
> > > >>> It is neccesary to set the maximum rate and limit the disk I/O
of
> > > >>>
> > > >>> VMs.
> > > >>>
> > > >>>
> > > >>>
> > > >>> Looking at the code I see you make no difference between Read
and
> > > >>>
> > > >>> Write
> > > >>>
> > > >>> IOps.
> > > >>>
> > > >>>
> > > >>> Qemu and libvirt support setting both a different rate for Read
and
> > > >>>
> > > >>> Write
> > > >>>
> > > >>> IOps which could benefit a lot of users.
> > > >>>
> > > >>>
> > > >>> It's also strange, in the polling side you collect both the Read
> and
> > > >>>
> > > >>> Write
> > > >>>
> > > >>> IOps, but on the throttling side you only go for a global value.
> > > >>>
> > > >>>
> > > >>> Write IOps are usually much more expensive then Read IOps, so
it
> > > >>>
> > > >>> seems
> > > >>>
> > > >>> like a valid use-case where that an admin would set a lower value
> for
> > > >>>
> > > >>> write
> > > >>>
> > > >>> IOps vs Read IOps.
> > > >>>
> > > >>>
> > > >>> Since this only supports KVM at this point I think it would be
of
> > > >>>
> > > >>> great
> > > >>>
> > > >>> value to at least have the mechanism in place to support both,
> > > >>>
> > > >>> implementing
> > > >>>
> > > >>> this later would be a lot of work.
> > > >>>
> > > >>>
> > > >>> If a hypervisor doesn't support setting different values for read
> and
> > > >>>
> > > >>> write you can always sum both up and set that as the total limit.
> > > >>>
> > > >>>
> > > >>> Can you explain why you implemented it this way?
> > > >>>
> > > >>>
> > > >>> Wido
> > > >>>
> > > >>>
> > > >>>  The feature includes:
> > > >>>
> > > >>>
> > > >>>
> > > >>> (1) set the maximum rate of VMs (in disk_offering, and global
> > > >>>
> > > >>> configuration)
> > > >>>
> > > >>> (2) change the maximum rate of VMs
> > > >>>
> > > >>> (3) limit the disk rate (total bps and iops)
> > > >>>
> > > >>> JIRA ticket: https://issues.apache.org/****
> > > >>>
> > > >>> jira/browse/CLOUDSTACK-1192<ht**tps://issues.apache.org/****
> > > >>>
> > > >>> jira/browse/CLOUDSTACK-1192<
> > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192>
> > > >>>
> > > >>> <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1192<
> > > >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192>
> > > >>>
> > > >>> <**
> > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192<
> > > >> https://issues.apache.org/jira/browse/CLOUDSTACK-1192>
> > > >>>
> > > >>>
> > > >>>
> > > >>> FS (I will update later) :
> > > >>>
> > > >>>
> > > >>>
> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
> > > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> > > >>>
> > > >>> <
> > > >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<
> > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
> > > >>>
> > > >>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****
> <
> > > >>> http://cwiki.apache.org/confluence/**>
> > > >>>
> > > >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
> > > >>>
> > > >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> <
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> Merge check list :-
> > > >>>
> > > >>>
> > > >>> * Did you check the branch's RAT execution success?
> > > >>>
> > > >>> Yes
> > > >>>
> > > >>>
> > > >>> * Are there new dependencies introduced?
> > > >>>
> > > >>> No
> > > >>>
> > > >>>
> > > >>> * What automated testing (unit and integration) is included in
the
> > > >>>
> > > >>> new
> > > >>>
> > > >>> feature?
> > > >>>
> > > >>> Unit tests are added.
> > > >>>
> > > >>>
> > > >>> * What testing has been done to check for potential regressions?
> > > >>>
> > > >>> (1) set the bytes rate and IOPS rate on CloudStack UI.
> > > >>>
> > > >>> (2) VM operations, including
> > > >>>
> > > >>> deploy, stop, start, reboot, destroy, expunge. migrate, restore
> > > >>>
> > > >>> (3) Volume operations, including
> > > >>>
> > > >>> Attach, Detach
> > > >>>
> > > >>>
> > > >>> To review the code, you can try
> > > >>>
> > > >>>  git diff c30057635d04a2396f84c588127d7e******be42e503a7
> > > >>>
> > > >>> f2e5591b710d04cc86815044f5823e******73a4a58944
> > > >>>
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Wei
> > > >>>
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > > >>>
> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******<
> > > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****>
> > > >>>
> > > >>> <
> > > >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**<
> > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**>
> > > >>>
> > > >>> VM+Disk+IO+Throttling<https://****cwiki.apache.org/confluence/****
> <
> > > >>> http://cwiki.apache.org/confluence/**>
> > > >>>
> > > >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling<https://cwiki.**
> > > >>>
> > > >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling
> <
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttling
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> [2] refs/heads/disk_io_throttling
> > > >>>
> > > >>> [3]
> > > >>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301<
> > > >> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301>
> > > >>>
> > > >>> <ht**tps://issues.apache.org/****jira/browse/CLOUDSTACK-1301<
> > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301>
> > > >>>
> > > >>> <ht**tps://issues.apache.org/**jira/**browse/CLOUDSTACK-1301<
> > > >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301>
> > > >>>
> > > >>> <**
> > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301<
> > > >> https://issues.apache.org/jira/browse/CLOUDSTACK-1301>
> > > >>>
> > > >>>
> > > >>>
> > > >>> <ht**tps://issues.apache.org/****jira/**browse/CLOUDSTACK-2071<
> > > >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071>
> > > >>>
> > > >>> **<
> > > >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071<
> > > >> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071>
> > > >>>
> > > >>> <**
> > > >>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071<
> > > >> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071>
> > > >>>
> > > >>> <h**ttps://issues.apache.org/jira/**browse/CLOUDSTACK-2071<
> > > >>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071>
> > > >>>
> > > >>>
> > > >>>
> > > >>> (**CLOUDSTACK-1301
> > > >>>
> > > >>> -     VM Disk I/O Throttling)
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> *Mike Tutkowski*
> > > >> *Senior CloudStack Developer, SolidFire Inc.*
> > > >> e: mike.tutkowski@solidfire.com
> > > >> o: 303.746.7302
> > > >> Advancing the way the world uses the
> > > >> cloud<http://solidfire.com/solution/overview/?video=play>
> > > >> *™*
> > > >>
> > >
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message