Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4082DC389 for ; Mon, 3 Jun 2013 01:15:29 +0000 (UTC) Received: (qmail 61035 invoked by uid 500); 3 Jun 2013 01:15:28 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 60949 invoked by uid 500); 3 Jun 2013 01:15:28 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 60941 invoked by uid 99); 3 Jun 2013 01:15:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 01:15:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.214.180 as permitted sender) Received: from [209.85.214.180] (HELO mail-ob0-f180.google.com) (209.85.214.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2013 01:15:20 +0000 Received: by mail-ob0-f180.google.com with SMTP id eh20so6111153obb.39 for ; Sun, 02 Jun 2013 18:14:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=tl9/XRd3dRGuIYLnk3ViF/eOOI2QBM7n4yQbFDvZalI=; b=Ei3/JgRYmIKzTPRkWEtflXsBxAbDqdmeRB65TqbPdamZIgVwNzRu5eORDvDoLWr53F L8Pqdt9WKGavBmnA1cPTJiHdWVfcqN5utCxRnL6FzNSUaizNwzGLKKBd4s+dBME3KREr bJ10F+kC6ts+BnfyWHcpW0P/koB37HJbZsl3d7AG/MUr/IfM00ydXdoOtAeyqHOZZK1Z ql1n7GkMhcFGW6GJSTsiO6mut1b3MMqpMvxegvpLyyoxKXi881XgIQD2PFZdj9GMHxxS Hw0BEaE55slAg783FVNUWBkzc9WnFdtV+/634418hC1xYEEkEywbuN1lGHeL3hDvv/nM 7HOQ== MIME-Version: 1.0 X-Received: by 10.182.84.135 with SMTP id z7mr9462839oby.35.1370222098641; Sun, 02 Jun 2013 18:14:58 -0700 (PDT) Received: by 10.182.10.66 with HTTP; Sun, 2 Jun 2013 18:14:58 -0700 (PDT) In-Reply-To: References: <51A899FE.7030800@widodh.nl> <51A8AAE2.4080808@widodh.nl> <9CFA8370-33D5-4A7D-9FB1-A14A1DB1B07B@basho.com> <51A8B0C8.2070904@widodh.nl> <51A9BFA4.9040903@widodh.nl> <-6717161953645650340@unknownmsgid> <7966164390304114442@unknownmsgid> Date: Sun, 2 Jun 2013 19:14:58 -0600 Message-ID: Subject: Re: [MERGE] disk_io_throttling to MASTER From: Mike Tutkowski To: John Burwell , "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=089e013a26967406d904de35b16f X-Gm-Message-State: ALoCoQk/gyWL8EqI1jWi9sJuQge2DFggVnkFiWz/c2BBnyRPhTzu0Fgds8lXiMMk4ToHDQGCVDn6 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a26967406d904de35b16f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Sending to the CS e-mail list (I accidentally only sent this to John before). On Sun, Jun 2, 2013 at 7:10 PM, Mike Tutkowski wrote: > Hi John, > > Let me see if I can answer your questions. > > 1) If we mean provide the ability to charge more for higher IOPS Disk > Offerings, then I would say 'yes.' > > 2) I might need more detail on what you're thinking here. > > 3) The IOPS in my patch code are from the storage system's point of view > only. If the hypervisor has a Max that is, say, below the Max of the > storage system, the storage system will never hit its Max. That being the > case, it probably makes sense to provide an error message to the user in > such a situation. I haven't actually seen Wei's code yet, but we should > coordinate on activities like this. > > > On Sun, Jun 2, 2013 at 6:54 PM, John Burwell wrote: > >> Mike, >> >> The things I want to understand are the following: >> >> 1) Is there value in capturing IOPS policies be captured in a common >> data model (e.g. for billing/usage purposes, expressing offerings). >> 2) Should there be a common interface model for reasoning about IOP >> provisioning at runtime? >> 3) How are conflicting provisioned IOPS configurations between a >> hypervisor and storage device reconciled? In particular, a scenario whe= re >> is lead to believe (and billed) for more IOPS configured for a VM than a >> storage device has been configured to deliver. Another scenario could a >> consistent configuration between a VM and a storage device at creation >> time, but a later modification to storage device introduces logical >> inconsistency. >> >> Thanks, >> -John >> >> On Jun 2, 2013, at 8:38 PM, Mike Tutkowski >> wrote: >> >> Hi John, >> >> I believe Wei's feature deals with controlling the max number of IOPS >> from the hypervisor side. >> >> My feature is focused on controlling IOPS from the storage system side. >> >> I hope that helps. :) >> >> >> On Sun, Jun 2, 2013 at 6:35 PM, John Burwell wrote: >> >>> Wei, >>> >>> My opinion is that no features should be merged until all functional >>> issues have been resolved and it is ready to turn over to test. Until = the >>> total Ops vs discrete read/write ops issue is addressed and re-reviewed= by >>> Wido, I don't think this criteria has been satisfied. >>> >>> Also, how does this work intersect/compliment the SolidFire patch ( >>> https://reviews.apache.org/r/11479/)? As I understand it that work is >>> also involves provisioned IOPS. I would like to ensure we don't have a >>> scenario where provisioned IOPS in KVM and SolidFire are unnecessarily >>> incompatible. >>> >>> Thanks, >>> -John >>> >>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU wrote: >>> >>> Wido, >>> >>> >>> Sure. I will change it next week. >>> >>> >>> -Wei >>> >>> >>> >>> 2013/6/1 Wido den Hollander >>> >>> >>> Hi Wei, >>> >>> >>> >>> On 06/01/2013 08:24 AM, Wei ZHOU wrote: >>> >>> >>> Wido, >>> >>> >>> Exactly. I have pushed the features into master. >>> >>> >>> If anyone object thems for technical reason till Monday, I will revert >>> >>> them. >>> >>> >>> For the sake of clarity I just want to mention again that we should >>> change >>> >>> the total IOps to R/W IOps asap so that we never release a version with >>> >>> only total IOps. >>> >>> >>> You laid the groundwork for the I/O throttling and that's great! We >>> should >>> >>> however prevent that we create legacy from day #1. >>> >>> >>> Wido >>> >>> >>> -Wei >>> >>> >>> >>> 2013/5/31 Wido den Hollander >>> >>> >>> On 05/31/2013 03:59 PM, John Burwell wrote: >>> >>> >>> Wido, >>> >>> >>> +1 -- this enhancement must to discretely support read and write IOPS. >>> >>> I >>> >>> don't see how it could be fixed later because I don't see how we >>> >>> correctly >>> >>> split total IOPS into read and write. Therefore, we would be stuck >>> >>> with a >>> >>> total unless/until we decided to break backwards compatibility. >>> >>> >>> >>> What Wei meant was merging it into master now so that it will go in the >>> >>> 4.2 branch and add Read / Write IOps before the 4.2 release so that 4.2 >>> >>> will be released with Read and Write instead of Total IOps. >>> >>> >>> This is to make the May 31st feature freeze date. But if the window mov= es >>> >>> (see other threads) then it won't be necessary to do that. >>> >>> >>> Wido >>> >>> >>> >>> I also completely agree that there is no association between network >>> >>> and >>> >>> >>> disk I/O. >>> >>> >>> Thanks, >>> >>> -John >>> >>> >>> On May 31, 2013, at 9:51 AM, Wido den Hollander wrote: >>> >>> >>> Hi Wei, >>> >>> >>> >>> On 05/31/2013 03:13 PM, Wei ZHOU wrote: >>> >>> >>> Hi Wido, >>> >>> >>> Thanks. Good question. >>> >>> >>> I thought about at the beginning. Finally I decided to ignore the >>> >>> difference of read and write mainly because the network throttling did >>> >>> not >>> >>> care the difference of sent and received bytes as well. >>> >>> That reasoning seems odd. Networking and disk I/O completely different. >>> >>> >>> Disk I/O is much more expensive in most situations then network >>> >>> bandwith. >>> >>> >>> Implementing it will be some copy-paste work. It could be >>> >>> implemented in >>> >>> >>> few days. For the deadline of feature freeze, I will implement it >>> >>> after >>> >>> that , if needed. >>> >>> >>> >>> It think it's a feature we can't miss. But if it goes into the 4.2 >>> >>> window we have to make sure we don't release with only total IOps and >>> >>> fix >>> >>> it in 4.3, that will confuse users. >>> >>> >>> Wido >>> >>> >>> -Wei >>> >>> >>> >>> >>> >>> >>> 2013/5/31 Wido den Hollander >>> >>> >>> Hi Wei, >>> >>> >>> >>> >>> On 05/30/2013 06:03 PM, Wei ZHOU wrote: >>> >>> >>> Hi, >>> >>> >>> I would like to merge disk_io_throttling branch into master. >>> >>> If nobody object, I will merge into master in 48 hours. >>> >>> The purpose is : >>> >>> >>> Virtual machines are running on the same storage device (local >>> >>> storage or >>> >>> share strage). Because of the rate limitation of device (such as >>> >>> iops), if >>> >>> one VM has large disk operation, it may affect the disk performance >>> >>> of >>> >>> other VMs running on the same storage device. >>> >>> It is neccesary to set the maximum rate and limit the disk I/O of >>> >>> VMs. >>> >>> >>> >>> Looking at the code I see you make no difference between Read and >>> >>> Write >>> >>> IOps. >>> >>> >>> Qemu and libvirt support setting both a different rate for Read and >>> >>> Write >>> >>> IOps which could benefit a lot of users. >>> >>> >>> It's also strange, in the polling side you collect both the Read and >>> >>> Write >>> >>> IOps, but on the throttling side you only go for a global value. >>> >>> >>> Write IOps are usually much more expensive then Read IOps, so it >>> >>> seems >>> >>> like a valid use-case where that an admin would set a lower value for >>> >>> write >>> >>> IOps vs Read IOps. >>> >>> >>> Since this only supports KVM at this point I think it would be of >>> >>> great >>> >>> value to at least have the mechanism in place to support both, >>> >>> implementing >>> >>> this later would be a lot of work. >>> >>> >>> If a hypervisor doesn't support setting different values for read and >>> >>> write you can always sum both up and set that as the total limit. >>> >>> >>> Can you explain why you implemented it this way? >>> >>> >>> Wido >>> >>> >>> The feature includes: >>> >>> >>> >>> (1) set the maximum rate of VMs (in disk_offering, and global >>> >>> configuration) >>> >>> (2) change the maximum rate of VMs >>> >>> (3) limit the disk rate (total bps and iops) >>> >>> JIRA ticket: https://issues.apache.org/**** >>> >>> jira/browse/CLOUDSTACK-1192>> >>> jira/browse/CLOUDSTACK-1192< >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192> >>> >>> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192> >>> >>> <** >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192 >>> >>> >>> >>> FS (I will update later) : >>> >>> >>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/****** >>> >>> < >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/** >>> >>> VM+Disk+IO+Throttling>> http://cwiki.apache.org/confluence/**> >>> >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling>> >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling< >>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throt= tling >>> > >>> >>> >>> >>> Merge check list :- >>> >>> >>> * Did you check the branch's RAT execution success? >>> >>> Yes >>> >>> >>> * Are there new dependencies introduced? >>> >>> No >>> >>> >>> * What automated testing (unit and integration) is included in the >>> >>> new >>> >>> feature? >>> >>> Unit tests are added. >>> >>> >>> * What testing has been done to check for potential regressions? >>> >>> (1) set the bytes rate and IOPS rate on CloudStack UI. >>> >>> (2) VM operations, including >>> >>> deploy, stop, start, reboot, destroy, expunge. migrate, restore >>> >>> (3) Volume operations, including >>> >>> Attach, Detach >>> >>> >>> To review the code, you can try >>> >>> git diff c30057635d04a2396f84c588127d7e******be42e503a7 >>> >>> f2e5591b710d04cc86815044f5823e******73a4a58944 >>> >>> >>> Best regards, >>> >>> Wei >>> >>> >>> [1] >>> >>> >>> https://cwiki.apache.org/******confluence/display/CLOUDSTACK/****** >>> >>> < >>> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/** >>> >>> VM+Disk+IO+Throttling>> http://cwiki.apache.org/confluence/**> >>> >>> display/CLOUDSTACK/VM+Disk+IO+****Throttling>> >>> apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling< >>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throt= tling >>> > >>> >>> >>> >>> [2] refs/heads/disk_io_throttling >>> >>> [3] >>> https://issues.apache.org/******jira/browse/CLOUDSTACK-1301 >>> >>> >> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301> >>> >>> >> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301> >>> >>> <** >>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301 >>> >>> >>> >>> >> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071> >>> >>> **< >>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071 >>> >>> <** >>> https://issues.apache.org/****jira/browse/CLOUDSTACK-2071 >>> >>> >> https://issues.apache.org/jira/browse/CLOUDSTACK-2071> >>> >>> >>> >>> (**CLOUDSTACK-1301 >>> >>> - VM Disk I/O Throttling) >>> >>> >>> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkowski@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> *=99* >> >> > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkowski@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > *=99* > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --089e013a26967406d904de35b16f--