Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3493DCB9B for ; Mon, 3 Jun 2013 17:22:07 +0000 (UTC) Received: (qmail 92457 invoked by uid 500); 3 Jun 2013 17:22:06 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 92404 invoked by uid 500); 3 Jun 2013 17:22:06 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 92394 invoked by uid 99); 3 Jun 2013 17:22:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 17:22:05 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-ob0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 17:21:58 +0000 Received: by mail-ob0-f172.google.com with SMTP id wo10so7723528obc.31 for ; Mon, 03 Jun 2013 10:21:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=P2jtapBYIIFdU5GPljWR6nuieq8P5gcMw9ZpK6it0MY=; b=mMLWjawLeCCxKsuWybLFcbTPciwEB2J4hn6YKh+qQ4zKOCeRwgK5pmv+MgSeZpTOFG YYCE6ksqFdu1yAh+Dad7BPXamZsTRxfZb+KtSvJGlBgqqP0E0TLQ05vmNFFitOxTOHvF 1rnPThoWqBWGgarfTFxDNUzTmBhEjuLKg0gNEQWqmRCymg5rN/Km0gIqks70tGNBxbqo ah0xjvYZ9MhwHZG9Y4AiYeYGkiME+oELlwT3u71/4Y2X5EgEzG91Q96GB7aI6foDBC8D D4+xax9IFigcJ60m/lWmZka08uD0r7YbQc0Pyj0MiTbAYIwYgDREE3Fzcpv6t673Z3sR iEEg== MIME-Version: 1.0 X-Received: by 10.60.62.162 with SMTP id z2mr10339015oer.140.1370280096981; Mon, 03 Jun 2013 10:21:36 -0700 (PDT) Received: by 10.182.10.66 with HTTP; Mon, 3 Jun 2013 10:21:36 -0700 (PDT) In-Reply-To: <1637190856.747092.1370279686542.JavaMail.root@bbits.ca> References: <-6717161953645650340@unknownmsgid> <7966164390304114442@unknownmsgid> <1637190856.747092.1370279686542.JavaMail.root@bbits.ca> Date: Mon, 3 Jun 2013 11:21:36 -0600 Message-ID: Subject: Re: [MERGE] disk_io_throttling to MASTER From: Mike Tutkowski To: "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=047d7b673d1a6d08e904de43324c X-Gm-Message-State: ALoCoQm9+lgPObbTY6f73VveojeEHj4bH+53oXXbf36EIVTnLjZQHvb9qvrDt1F4GuScmCahPNpd X-Virus-Checked: Checked by ClamAV on apache.org --047d7b673d1a6d08e904de43324c Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Yes, ultimately I would like to support all hypervisors that CloudStack supports. I think I'm just out of time for 4.2 to get KVM in. Right now this plug-in supports XenServer. Depending on what we do with regards to 4.2 feature freeze, I have it working for VMware in my sandbox, as well. Also, just to be clear, this is all in regards to Disk Offerings. I plan to support Compute Offerings post 4.2. On Mon, Jun 3, 2013 at 11:14 AM, Kelcey Jamison Damage wro= te: > Is there any plan on supporting KVM in the patch cycle post 4.2? > > ----- Original Message ----- > From: "Mike Tutkowski" > To: dev@cloudstack.apache.org > Sent: Monday, June 3, 2013 10:12:32 AM > Subject: Re: [MERGE] disk_io_throttling to MASTER > > I agree on merging Wei's feature first, then mine. > > If his feature is for KVM only, then it is a non issue as I don't support > KVM in 4.2. > > > On Mon, Jun 3, 2013 at 8:55 AM, Wei ZHOU wrote: > > > John, > > > > For the billing, as no one works on billing now, users need to calculat= e > > the billing by themselves. They can get the service_offering and > > disk_offering of a VMs and volumes for calculation. Of course it is > better > > to tell user the exact limitation value of individual volume, and netwo= rk > > rate limitation for nics as well. I can work on it later. Do you think = it > > is a part of I/O throttling? > > > > Sorry my misunstand the second the question. > > > > Agree with what you said about the two features. > > > > -Wei > > > > > > 2013/6/3 John Burwell > > > > > Wei, > > > > > > > > > On Jun 3, 2013, at 2:13 AM, Wei ZHOU wrote: > > > > > > > Hi John, Mike > > > > > > > > I hope Mike's aswer helps you. I am trying to adding more. > > > > > > > > (1) I think billing should depend on IO statistics rather than IOPS > > > > limitation. Please review disk_io_stat if you have time. > disk_io_stat > > > can > > > > get the IO statistics including bytes/iops read/write for an > individual > > > > virtual machine. > > > > > > Going by the AWS model, customers are billed more for volumes with > > > provisioned IOPS, as well as, for those operations ( > > > http://aws.amazon.com/ebs/). I would imagine our users would like th= e > > > option to employ similar cost models. Could an operator implement > such a > > > billing model in the current patch? > > > > > > > > > > > (2) Do you mean IOPS runtime change? KVM supports setting IOPS/BPS > > > > limitation for a running virtual machine through command line. > However, > > > > CloudStack does not support changing the parameters of a created > > offering > > > > (computer offering or disk offering). > > > > > > I meant at the Java interface level. I apologize for being unclear. > Can > > > we more generalize allocation algorithms with a set of interfaces tha= t > > > describe the service guarantees provided by a resource? > > > > > > > > > > > (3) It is a good question. Maybe it is better to commit Mike's patc= h > > > after > > > > disk_io_throttling as Mike needs to consider the limitation in > > hypervisor > > > > type, I think. > > > > > > I will expand on my thoughts in a later response to Mike regarding th= e > > > touch points between these two features. I think that > disk_io_throttling > > > will need to be merged before SolidFire, but I think we need closer > > > coordination between the branches (possibly have have solidfire track > > > disk_io_throttling) to coordinate on this issue. > > > > > > > > > > > - Wei > > > > > > > > > > > > 2013/6/3 John Burwell > > > > > > > >> Mike, > > > >> > > > >> The things I want to understand are the following: > > > >> > > > >> 1) Is there value in capturing IOPS policies be captured in a > common > > > >> data model (e.g. for billing/usage purposes, expressing offerings)= . > > > >> 2) Should there be a common interface model for reasoning about > IOP > > > >> provisioning at runtime? > > > >> 3) How are conflicting provisioned IOPS configurations between = a > > > >> hypervisor and storage device reconciled? In particular, a scenar= io > > > where > > > >> is lead to believe (and billed) for more IOPS configured for a VM > > than a > > > >> storage device has been configured to deliver. Another scenario > > could a > > > >> consistent configuration between a VM and a storage device at > creation > > > >> time, but a later modification to storage device introduces logica= l > > > >> inconsistency. > > > >> > > > >> Thanks, > > > >> -John > > > >> > > > >> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski < > > > mike.tutkowski@solidfire.com> > > > >> wrote: > > > >> > > > >> Hi John, > > > >> > > > >> I believe Wei's feature deals with controlling the max number of > IOPS > > > from > > > >> the hypervisor side. > > > >> > > > >> My feature is focused on controlling IOPS from the storage system > > side. > > > >> > > > >> I hope that helps. :) > > > >> > > > >> > > > >> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell > > > wrote: > > > >> > > > >>> Wei, > > > >>> > > > >>> My opinion is that no features should be merged until all > functional > > > >>> issues have been resolved and it is ready to turn over to test. > > Until > > > >> the > > > >>> total Ops vs discrete read/write ops issue is addressed and > > re-reviewed > > > >> by > > > >>> Wido, I don't think this criteria has been satisfied. > > > >>> > > > >>> Also, how does this work intersect/compliment the SolidFire patch= ( > > > >>> https://reviews.apache.org/r/11479/)? As I understand it that > work > > is > > > >>> also involves provisioned IOPS. I would like to ensure we don't > > have a > > > >>> scenario where provisioned IOPS in KVM and SolidFire are > > unnecessarily > > > >>> incompatible. > > > >>> > > > >>> Thanks, > > > >>> -John > > > >>> > > > >>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU > wrote: > > > >>> > > > >>> Wido, > > > >>> > > > >>> > > > >>> Sure. I will change it next week. > > > >>> > > > >>> > > > >>> -Wei > > > >>> > > > >>> > > > >>> > > > >>> 2013/6/1 Wido den Hollander > > > >>> > > > >>> > > > >>> Hi Wei, > > > >>> > > > >>> > > > >>> > > > >>> On 06/01/2013 08:24 AM, Wei ZHOU wrote: > > > >>> > > > >>> > > > >>> Wido, > > > >>> > > > >>> > > > >>> Exactly. I have pushed the features into master. > > > >>> > > > >>> > > > >>> If anyone object thems for technical reason till Monday, I will > > revert > > > >>> > > > >>> them. > > > >>> > > > >>> > > > >>> For the sake of clarity I just want to mention again that we shou= ld > > > >> change > > > >>> > > > >>> the total IOps to R/W IOps asap so that we never release a versio= n > > with > > > >>> > > > >>> only total IOps. > > > >>> > > > >>> > > > >>> You laid the groundwork for the I/O throttling and that's great! = We > > > >> should > > > >>> > > > >>> however prevent that we create legacy from day #1. > > > >>> > > > >>> > > > >>> Wido > > > >>> > > > >>> > > > >>> -Wei > > > >>> > > > >>> > > > >>> > > > >>> 2013/5/31 Wido den Hollander > > > >>> > > > >>> > > > >>> On 05/31/2013 03:59 PM, John Burwell wrote: > > > >>> > > > >>> > > > >>> Wido, > > > >>> > > > >>> > > > >>> +1 -- this enhancement must to discretely support read and write > > IOPS. > > > >>> > > > >>> I > > > >>> > > > >>> don't see how it could be fixed later because I don't see how we > > > >>> > > > >>> correctly > > > >>> > > > >>> split total IOPS into read and write. Therefore, we would be stu= ck > > > >>> > > > >>> with a > > > >>> > > > >>> total unless/until we decided to break backwards compatibility. > > > >>> > > > >>> > > > >>> > > > >>> What Wei meant was merging it into master now so that it will go = in > > the > > > >>> > > > >>> 4.2 branch and add Read / Write IOps before the 4.2 release so th= at > > 4.2 > > > >>> > > > >>> will be released with Read and Write instead of Total IOps. > > > >>> > > > >>> > > > >>> This is to make the May 31st feature freeze date. But if the wind= ow > > > moves > > > >>> > > > >>> (see other threads) then it won't be necessary to do that. > > > >>> > > > >>> > > > >>> Wido > > > >>> > > > >>> > > > >>> > > > >>> I also completely agree that there is no association between > network > > > >>> > > > >>> and > > > >>> > > > >>> > > > >>> disk I/O. > > > >>> > > > >>> > > > >>> Thanks, > > > >>> > > > >>> -John > > > >>> > > > >>> > > > >>> On May 31, 2013, at 9:51 AM, Wido den Hollander > > > wrote: > > > >>> > > > >>> > > > >>> Hi Wei, > > > >>> > > > >>> > > > >>> > > > >>> On 05/31/2013 03:13 PM, Wei ZHOU wrote: > > > >>> > > > >>> > > > >>> Hi Wido, > > > >>> > > > >>> > > > >>> Thanks. Good question. > > > >>> > > > >>> > > > >>> I thought about at the beginning. Finally I decided to ignore th= e > > > >>> > > > >>> difference of read and write mainly because the network throttlin= g > > did > > > >>> > > > >>> not > > > >>> > > > >>> care the difference of sent and received bytes as well. > > > >>> > > > >>> That reasoning seems odd. Networking and disk I/O completely > > different. > > > >>> > > > >>> > > > >>> Disk I/O is much more expensive in most situations then network > > > >>> > > > >>> bandwith. > > > >>> > > > >>> > > > >>> Implementing it will be some copy-paste work. It could be > > > >>> > > > >>> implemented in > > > >>> > > > >>> > > > >>> few days. For the deadline of feature freeze, I will implement it > > > >>> > > > >>> after > > > >>> > > > >>> that , if needed. > > > >>> > > > >>> > > > >>> > > > >>> It think it's a feature we can't miss. But if it goes into the 4.= 2 > > > >>> > > > >>> window we have to make sure we don't release with only total IOps > and > > > >>> > > > >>> fix > > > >>> > > > >>> it in 4.3, that will confuse users. > > > >>> > > > >>> > > > >>> Wido > > > >>> > > > >>> > > > >>> -Wei > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> 2013/5/31 Wido den Hollander > > > >>> > > > >>> > > > >>> Hi Wei, > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On 05/30/2013 06:03 PM, Wei ZHOU wrote: > > > >>> > > > >>> > > > >>> Hi, > > > >>> > > > >>> > > > >>> I would like to merge disk_io_throttling branch into master. > > > >>> > > > >>> If nobody object, I will merge into master in 48 hours. > > > >>> > > > >>> The purpose is : > > > >>> > > > >>> > > > >>> Virtual machines are running on the same storage device (local > > > >>> > > > >>> storage or > > > >>> > > > >>> share strage). Because of the rate limitation of device (such as > > > >>> > > > >>> iops), if > > > >>> > > > >>> one VM has large disk operation, it may affect the disk performan= ce > > > >>> > > > >>> of > > > >>> > > > >>> other VMs running on the same storage device. > > > >>> > > > >>> It is neccesary to set the maximum rate and limit the disk I/O of > > > >>> > > > >>> VMs. > > > >>> > > > >>> > > > >>> > > > >>> Looking at the code I see you make no difference between Read and > > > >>> > > > >>> Write > > > >>> > > > >>> IOps. > > > >>> > > > >>> > > > >>> Qemu and libvirt support setting both a different rate for Read a= nd > > > >>> > > > >>> Write > > > >>> > > > >>> IOps which could benefit a lot of users. > > > >>> > > > >>> > > > >>> It's also strange, in the polling side you collect both the Read > and > > > >>> > > > >>> Write > > > >>> > > > >>> IOps, but on the throttling side you only go for a global value. > > > >>> > > > >>> > > > >>> Write IOps are usually much more expensive then Read IOps, so it > > > >>> > > > >>> seems > > > >>> > > > >>> like a valid use-case where that an admin would set a lower value > for > > > >>> > > > >>> write > > > >>> > > > >>> IOps vs Read IOps. > > > >>> > > > >>> > > > >>> Since this only supports KVM at this point I think it would be of > > > >>> > > > >>> great > > > >>> > > > >>> value to at least have the mechanism in place to support both, > > > >>> > > > >>> implementing > > > >>> > > > >>> this later would be a lot of work. > > > >>> > > > >>> > > > >>> If a hypervisor doesn't support setting different values for read > and > > > >>> > > > >>> write you can always sum both up and set that as the total limit. > > > >>> > > > >>> > > > >>> Can you explain why you implemented it this way? > > > >>> > > > >>> > > > >>> Wido > > > >>> > > > >>> > > > >>> The feature includes: > > > >>> > > > >>> > > > >>> > > > >>> (1) set the maximum rate of VMs (in disk_offering, and global > > > >>> > > > >>> configuration) > > > >>> > > > >>> (2) change the maximum rate of VMs > > > >>> > > > >>> (3) limit the disk rate (total bps and iops) > > > >>> > > > >>> JIRA ticket: https://issues.apache.org/**** > > > >>> > > > >>> jira/browse/CLOUDSTACK-1192 > > >>> > > > >>> jira/browse/CLOUDSTACK-1192< > > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192> > > > >>> > > > >>> > > >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192> > > > >>> > > > >>> <** > > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192< > > > >> https://issues.apache.org/jira/browse/CLOUDSTACK-1192> > > > >>> > > > >>> > > > >>> > > > >>> FS (I will update later) : > > > >>> > > > >>> > > > >>> > https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******< > > > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****> > > > >>> > > > >>> < > > > >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**< > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**> > > > >>> > > > >>> VM+Disk+IO+Throttling < > > > >>> http://cwiki.apache.org/confluence/**> > > > >>> > > > >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling > > >>> > > > >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttlin= g > < > > > >>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttl= ing > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> Merge check list :- > > > >>> > > > >>> > > > >>> * Did you check the branch's RAT execution success? > > > >>> > > > >>> Yes > > > >>> > > > >>> > > > >>> * Are there new dependencies introduced? > > > >>> > > > >>> No > > > >>> > > > >>> > > > >>> * What automated testing (unit and integration) is included in th= e > > > >>> > > > >>> new > > > >>> > > > >>> feature? > > > >>> > > > >>> Unit tests are added. > > > >>> > > > >>> > > > >>> * What testing has been done to check for potential regressions? > > > >>> > > > >>> (1) set the bytes rate and IOPS rate on CloudStack UI. > > > >>> > > > >>> (2) VM operations, including > > > >>> > > > >>> deploy, stop, start, reboot, destroy, expunge. migrate, restore > > > >>> > > > >>> (3) Volume operations, including > > > >>> > > > >>> Attach, Detach > > > >>> > > > >>> > > > >>> To review the code, you can try > > > >>> > > > >>> git diff c30057635d04a2396f84c588127d7e******be42e503a7 > > > >>> > > > >>> f2e5591b710d04cc86815044f5823e******73a4a58944 > > > >>> > > > >>> > > > >>> Best regards, > > > >>> > > > >>> Wei > > > >>> > > > >>> > > > >>> [1] > > > >>> > > > >>> > > > >>> > https://cwiki.apache.org/******confluence/display/CLOUDSTACK/******< > > > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****> > > > >>> > > > >>> < > > > >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/**< > > > >> https://cwiki.apache.org/**confluence/display/CLOUDSTACK/**> > > > >>> > > > >>> VM+Disk+IO+Throttling < > > > >>> http://cwiki.apache.org/confluence/**> > > > >>> > > > >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling > > >>> > > > >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttlin= g > < > > > >>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttl= ing > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> [2] refs/heads/disk_io_throttling > > > >>> > > > >>> [3] > > > >>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301< > > > >> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301> > > > >>> > > > >>> > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301> > > > >>> > > > >>> > > >>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301> > > > >>> > > > >>> <** > > > >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301< > > > >> https://issues.apache.org/jira/browse/CLOUDSTACK-1301> > > > >>> > > > >>> > > > >>> > > > >>> > > >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071> > > > >>> > > > >>> **< > > > >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071< > > > >> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071> > > > >>> > > > >>> <** > > > >>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071< > > > >> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071> > > > >>> > > > >>> > > >>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071> > > > >>> > > > >>> > > > >>> > > > >>> (**CLOUDSTACK-1301 > > > >>> > > > >>> - VM Disk I/O Throttling) > > > >>> > > > >>> > > > >>> > > > >> > > > >> > > > >> -- > > > >> *Mike Tutkowski* > > > >> *Senior CloudStack Developer, SolidFire Inc.* > > > >> e: mike.tutkowski@solidfire.com > > > >> o: 303.746.7302 > > > >> Advancing the way the world uses the > > > >> cloud > > > >> *=99* > > > >> > > > > > > > > > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkowski@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud > *=99* > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --047d7b673d1a6d08e904de43324c--