Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 28AFC107DC for ; Tue, 4 Jun 2013 21:29:15 +0000 (UTC) Received: (qmail 5556 invoked by uid 500); 4 Jun 2013 21:29:14 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 5507 invoked by uid 500); 4 Jun 2013 21:29:14 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 5499 invoked by uid 99); 4 Jun 2013 21:29:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jun 2013 21:29:14 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.214.174 as permitted sender) Received: from [209.85.214.174] (HELO mail-ob0-f174.google.com) (209.85.214.174) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jun 2013 21:28:55 +0000 Received: by mail-ob0-f174.google.com with SMTP id wd20so1334570obb.5 for ; Tue, 04 Jun 2013 14:28:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=6p02a5xri74omqmRYgzlEGP9sjjEsDgf8eJo2qFYIi8=; b=oEYhcY6r8N6qFtXgIqZp5+o6Ur5yZxn9RqCx+1FPf8yqoztetm2W7S5ZV9Wv/HExZd uWeWQ4C6Jn5k3Bc/tuNrRWiVtx4tE1SMqH32ykIiIfneYiGLnkW0L/uzIbkkowe4cwkr o8wdVwvgYRSo10vpQA52puUZTGAYSatPBHf2MQ9L0TfdK1LRNXzjzcfLTY/WfAxRMsZw PX6tq3n5CyLDBzCfFgRnWowbcRMdVOengZfXwbji9YonomZjF3RO85EacpGUJH5s1ZDz wm7FgwQwlarY+s/p6UnaEbUrYIW2G6Wxj7/wutsq9WGY0mZJd0vOffSYlEGZEc8tnXtl UlxA== MIME-Version: 1.0 X-Received: by 10.60.124.100 with SMTP id mh4mr13487239oeb.122.1370381312759; Tue, 04 Jun 2013 14:28:32 -0700 (PDT) Received: by 10.182.10.66 with HTTP; Tue, 4 Jun 2013 14:28:32 -0700 (PDT) In-Reply-To: <33D19C43-D344-462C-8E32-DF27E20E7D53@basho.com> References: <-6717161953645650340@unknownmsgid> <7966164390304114442@unknownmsgid> <1637190856.747092.1370279686542.JavaMail.root@bbits.ca> <93B5A79D-CDC1-4D3B-9742-16CE018454BA@basho.com> <390BD462-1751-4FFE-B237-CDD19D2DCEA7@basho.com> <2CDCC854-9144-48A2-ABA7-8D11B239BCA4@basho.com> <33D19C43-D344-462C-8E32-DF27E20E7D53@basho.com> Date: Tue, 4 Jun 2013 15:28:32 -0600 Message-ID: Subject: Re: [MERGE] disk_io_throttling to MASTER From: Mike Tutkowski To: "dev@cloudstack.apache.org" Content-Type: multipart/alternative; boundary=047d7b5d300c5a9fac04de5ac3f4 X-Gm-Message-State: ALoCoQkA0HQ6KsHWEw/IXVlBPF2H2JJ7YLlvhsqm41EZiL/It2DvZz1WF2iBasz0is/WfauR4eqj X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5d300c5a9fac04de5ac3f4 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable "So, in essence, the SolidFire plugin introduces the notion of a managed iSCSI device and provisioned IOPS." Technically, the SolidFire plug-in just introduces the notion of provisioned storage IOPS. The storage framework that leverages the plug-in was incomplete, so I had to try to add in the notion of a managed iSCSI device. I appreciate all the time you've been spending on this. :) Do you have a recommendation as to how we should accomplish what you're looking for? Thanks! On Tue, Jun 4, 2013 at 3:19 PM, John Burwell wrote: > Mike, > > So, in essence, the SolidFire plugin introduces the notion of a managed > iSCSI device and provisioned IOPS. > > I want to see a separation of the management capabilities (i.e. can a > device be managed/does an operator want it managed by CloudStack) from th= e > storage protocol. Ideally, we should end up with a semantic that will > allow any type of storage device to be managed. I also want to make > progress on decoupling the storage types from the hypervisor definitions. > > Thanks, > -John > > On Jun 4, 2013, at 5:13 PM, Mike Tutkowski > wrote: > > > Hi John, > > > > No problem. Answers are below in red. > > > > Thanks! > > > > > > On Tue, Jun 4, 2013 at 2:55 PM, John Burwell wrote= : > > > >> Mike, > >> > >> Could you please answer the following questions for me with regards to > the > >> operation of the SolidFire plugin: > >> > >> What is the cardinality between iSCSI LUNs and SAN volumes? > >> > > Each SAN volume is equivalent to a single LUN (LUN 0). > > 1 SAN volume : 1 LUN > > > >> What is the cardinality between SAN Volumes and CloudStack Volumes? > >> > > 1 SAN volume : 1 CloudStack volume > > > >> Are the LUN(s) created by the management server or externally by the > >> operator? > >> > > When used with the SolidFire plug-in, a SAN volume (same as a SAN LUN) = is > > created by the management server (via the plug-in) the first time the > > CloudStack volume is attached to a hypervisor. > > > > If you don't want to use the SolidFire plug-in, but still want to use a > > SolidFire volume (LUN), you can do this already today (prior to 4.2). T= he > > admin manually creates the SAN volume and - in this case - multiple VMs > and > > data disks can share this SAN volume. While you can do this today, it i= s > > not useful if you want to enforce storage QoS. > > > >> Are the SAN volumes by the management server or externally by the > operator? > >> > > When the SolidFire plug-in is used, the SAN volumes are completely > managed > > by the management server (via the plug-in). There is no admin > interaction. > > This allows for a 1:1 mapping between a SAN volume and a CloudStack > volume, > > which is necessary for any storage vendor that supports true QoS. > > > >> > >> I would like to clarify how these pieces are related and expected to > >> operate. > >> > >> Thanks, > >> -John > >> > >> On Jun 4, 2013, at 3:46 PM, Mike Tutkowski < > mike.tutkowski@solidfire.com> > >> wrote: > >> > >>> "In particular, how do we ensure that multiple VMs with provisioned > IOPS > >>> won't be cut off by the underlying storage." > >>> > >>> In the storage QoS world, we need to map a single SAN volume (LUN) to= a > >>> single CloudStack volume. We cannot have multiple CloudStack volumes > >>> sharing a single SAN volume and still guarantee QoS. > >>> > >>> If the user wants to have a single SAN volume house more than one > >>> CloudStack volume, then can do that today without any of my plug-in > code. > >>> > >>> > >>> On Tue, Jun 4, 2013 at 1:43 PM, Mike Tutkowski < > >> mike.tutkowski@solidfire.com > >>>> wrote: > >>> > >>>> "The administrator will allocate a SAN volume for CloudStack's use > onto > >>>> which CloudStack volumes will be created." > >>>> > >>>> I think we crossed e-mails. :) > >>>> > >>>> Check out my recent e-mail on this. > >>>> > >>>> > >>>> On Tue, Jun 4, 2013 at 1:41 PM, John Burwell > >> wrote: > >>>> > >>>>> Mike, > >>>>> > >>>>> You are coming to the part which concerns me -- concepts from the > >>>>> hypervisor are leaking into storage layer. > >>>>> > >>>>> Thanks, > >>>>> -John > >>>>> > >>>>> On Jun 4, 2013, at 3:35 PM, Mike Tutkowski < > >> mike.tutkowski@solidfire.com> > >>>>> wrote: > >>>>> > >>>>>> The weird part is that the iSCSI type is today only used (as far a= s > I > >>>>> know) > >>>>>> in regards to XenServer (when you have not PreSetup an SR). > >>>>>> > >>>>>> If you want to use your iSCSI volume from VMware, it uses the vmfs > >> type. > >>>>>> > >>>>>> If you want to use your iSCSI volume from KVM, it uses the > >>>>> SharedMountPoint > >>>>>> type. > >>>>>> > >>>>>> So, I suppose mine and Edison's thinking here was to make a new ty= pe > >> of > >>>>>> storage to describe this dynamic ability Edison added into the > storage > >>>>>> framework. Maybe it should be more specificy, though: Dynamic_iSCS= I > >>>>> versus, > >>>>>> say, Dynamic_FC. > >>>>>> > >>>>>> > >>>>>> On Tue, Jun 4, 2013 at 1:27 PM, Mike Tutkowski < > >>>>> mike.tutkowski@solidfire.com > >>>>>>> wrote: > >>>>>> > >>>>>>> "The storage device itself shouldn't know or care that it is bein= g > >> used > >>>>>>> for a Xen SR -- simply be able to answer questions about it is > >>>>> storing." > >>>>>>> > >>>>>>> I see...so your concern here is that the SolidFire plug-in needs = to > >>>>> call > >>>>>>> itself "Dynamic" storage so that the hypervisor logic knows to > treat > >>>>> it as > >>>>>>> such. > >>>>>>> > >>>>>>> I'm totally open to removing that constraint and just calling it > >> iSCSI > >>>>> or > >>>>>>> whatever. We would just need a way for the hypervisor attach logi= c > to > >>>>>>> detect this new requirement and perform the necessary activities. > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Jun 4, 2013 at 1:24 PM, John Burwell > >>>>> wrote: > >>>>>>> > >>>>>>>> Mike, > >>>>>>>> > >>>>>>>> See my responses in-line. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> -John > >>>>>>>> > >>>>>>>> On Jun 4, 2013, at 3:10 PM, Mike Tutkowski < > >>>>> mike.tutkowski@solidfire.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I'm trying to picture this: > >>>>>>>>> > >>>>>>>>> "Finally, while CloudStack may be able to manage a device, an > >>>>> operator > >>>>>>>> may > >>>>>>>>> chose to leave it unmanaged by CloudStack (e.g. the device is > >> shared > >>>>> by > >>>>>>>>> many services, and the operator has chosen to dedicate only a > >> portion > >>>>>>>> of it > >>>>>>>>> to CloudStack). Does my reasoning make sense?" > >>>>>>>>> > >>>>>>>>> I guess I'm not sure how creating a SAN volume via the plug-in > >>>>> (before > >>>>>>>> an > >>>>>>>>> attach request to the hypervisor) would work unless the > hypervisor > >>>>>>>> consumes > >>>>>>>>> the SAN volume in the form of, say, an SR. > >>>>>>>> > >>>>>>>> My thinking is that, independent of CloudStack, an operator > >> allocates > >>>>> a > >>>>>>>> chunk of a SAN to CloudStack, and exposes it through a LUN. Th= ey > >>>>> simply > >>>>>>>> want to turn control of that LUN over to CloudStack, but not all= ow > >>>>>>>> CloudStack to allocate anymore LUNs. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> As the attach logic stands prior to my changes, we would be > passing > >>>>> in a > >>>>>>>>> SAN volume that does not have the necessary hypervisor support > >> (like > >>>>> an > >>>>>>>> SR) > >>>>>>>>> and the logic will fail. > >>>>>>>>> > >>>>>>>>> Are you thinking we should maybe have the storage framework > itself > >>>>>>>> detect > >>>>>>>>> that such a SAN volume needs support from the hypervisor side a= nd > >>>>> have > >>>>>>>> it > >>>>>>>>> call into the agent code specifically to create the SR before t= he > >>>>> attach > >>>>>>>>> logic runs in the agent code? > >>>>>>>> > >>>>>>>> I think the hypervisor management plugin should have a rich enou= gh > >>>>>>>> interface to storage to determine available for volume storage. > For > >>>>> Xen, > >>>>>>>> this interface would allow the interrogation of the device to > >>>>> determine the > >>>>>>>> SR is present. The storage device itself shouldn't know or car= e > >>>>> that it > >>>>>>>> is being used for a Xen SR -- simply be able to answer questions > >>>>> about it > >>>>>>>> is storing. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, Jun 4, 2013 at 1:01 PM, Mike Tutkowski < > >>>>>>>> mike.tutkowski@solidfire.com > >>>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> So, the flow is as follows: > >>>>>>>>>> > >>>>>>>>>> * The admin registers the SolidFire driver (which is a type of > >>>>>>>> so-called > >>>>>>>>>> Dynamic storage). Once this is done, a new Primary Storage sho= ws > >> up > >>>>> in > >>>>>>>> the > >>>>>>>>>> applicable zone. > >>>>>>>>>> > >>>>>>>>>> * The admin creates a Disk Offering that references the storag= e > >> tag > >>>>> of > >>>>>>>> the > >>>>>>>>>> newly created Primary Storage. > >>>>>>>>>> > >>>>>>>>>> * The end user creates a CloudStack volume. This leads to a ne= w > >> row > >>>>> in > >>>>>>>> the > >>>>>>>>>> cloud.volumes table. > >>>>>>>>>> > >>>>>>>>>> * The end user attaches the CloudStack volume to a VM (attach > >> disk). > >>>>>>>> This > >>>>>>>>>> leads to the storage framework calling the plug-in to create a > new > >>>>>>>> volume > >>>>>>>>>> on its storage system (in my case, a SAN). The plug-in also > >> updates > >>>>> the > >>>>>>>>>> cloud.volumes row with applicable data (like the IQN of the SA= N > >>>>>>>> volume). > >>>>>>>>>> This plug-in code is only invoked if the CloudStack volume is = in > >> the > >>>>>>>>>> 'Allocated' state. After the attach, the volume will be in the > >>>>> 'Ready' > >>>>>>>>>> state (even after a detach disk) and the plug-in code will not > be > >>>>>>>> called > >>>>>>>>>> again to create this SAN volume. > >>>>>>>>>> > >>>>>>>>>> * The hypervisor-attach logic is run and detects the CloudStac= k > >>>>> volume > >>>>>>>> to > >>>>>>>>>> attach needs "assistance" in the form of a hypervisor data > >> structure > >>>>>>>> (ex. > >>>>>>>>>> an SR on XenServer). > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Jun 4, 2013 at 12:54 PM, Mike Tutkowski < > >>>>>>>>>> mike.tutkowski@solidfire.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> "To ensure that we are in sync on terminology, volume, in the= se > >>>>>>>>>>> definitions, refers to the physical allocation on the device, > >>>>>>>> correct?" > >>>>>>>>>>> > >>>>>>>>>>> Yes...when I say 'volume', I try to mean 'SAN volume'. > >>>>>>>>>>> > >>>>>>>>>>> To refer to the 'volume' the end user can make in CloudStack,= I > >>>>> try to > >>>>>>>>>>> use 'CloudStack volume'. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jun 4, 2013 at 12:50 PM, Mike Tutkowski < > >>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi John, > >>>>>>>>>>>> > >>>>>>>>>>>> What you say here may very well make sense, but I'm having a > >> hard > >>>>>>>> time > >>>>>>>>>>>> envisioning it. > >>>>>>>>>>>> > >>>>>>>>>>>> Perhaps we should draw Edison in on this conversation as he > was > >>>>> the > >>>>>>>>>>>> initial person to suggest the approach I took. > >>>>>>>>>>>> > >>>>>>>>>>>> What do you think? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Jun 4, 2013 at 12:42 PM, John Burwell < > >> jburwell@basho.com > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Mike, > >>>>>>>>>>>>> > >>>>>>>>>>>>> It feels like we are combining two distinct concepts -- > storage > >>>>>>>> device > >>>>>>>>>>>>> management and storage protocols. In both cases, we are > >>>>>>>> communicating with > >>>>>>>>>>>>> ISCSI, but one allows the system to create/delete volumes > >>>>> (Dynamic) > >>>>>>>> on the > >>>>>>>>>>>>> device while the other requires the volume to be volume to = be > >>>>>>>> managed > >>>>>>>>>>>>> outside of the CloudStack context. To ensure that we are i= n > >>>>> sync on > >>>>>>>>>>>>> terminology, volume, in these definitions, refers to the > >> physical > >>>>>>>>>>>>> allocation on the device, correct? Minimally, we must be > able > >> to > >>>>>>>>>>>>> communicate with a storage device to move bits from one pla= ce > >> to > >>>>>>>> another, > >>>>>>>>>>>>> read bits, delete bits, etc. Optionally, a storage device > may > >> be > >>>>>>>> able to > >>>>>>>>>>>>> managed by CloudStack. Therefore, we can have a unmanaged > >> iSCSI > >>>>>>>> device > >>>>>>>>>>>>> onto which we store a Xen SR, and we can have a managed > >> SolidFire > >>>>>>>> iSCSI > >>>>>>>>>>>>> device on which CloudStack is capable of allocating LUNs an= d > >>>>> storing > >>>>>>>>>>>>> volumes. Finally, while CloudStack may be able to manage a > >>>>> device, > >>>>>>>> an > >>>>>>>>>>>>> operator may chose to leave it unmanaged by CloudStack (e.g= . > >> the > >>>>>>>> device is > >>>>>>>>>>>>> shared by many services, and the operator has chosen to > >> dedicate > >>>>>>>> only a > >>>>>>>>>>>>> portion of it to CloudStack). Does my reasoning make sense= ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Assuming my thoughts above are reasonable, it seems > appropriate > >>>>> to > >>>>>>>>>>>>> strip the management concerns from StoragePoolType, add the > >>>>> notion > >>>>>>>> of a > >>>>>>>>>>>>> storage device with an attached driver that indicates wheth= er > >> or > >>>>>>>> not is > >>>>>>>>>>>>> managed by CloudStack, and establish a abstraction > >> representing a > >>>>>>>> physical > >>>>>>>>>>>>> allocation on a device separate that is associated with a > >> volume. > >>>>>>>> With > >>>>>>>>>>>>> these notions in place, hypervisor drivers can declare whic= h > >>>>>>>> protocols they > >>>>>>>>>>>>> support and when they encounter a device managed by > CloudStack, > >>>>>>>> utilize the > >>>>>>>>>>>>> management operations exposed by the driver to automate > >>>>> allocation. > >>>>>>>> If > >>>>>>>>>>>>> these thoughts/concepts make sense, then we can sit down an= d > >>>>> drill > >>>>>>>> down to > >>>>>>>>>>>>> a more detailed design. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> -John > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Jun 3, 2013, at 5:25 PM, Mike Tutkowski < > >>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Here is the difference between the current iSCSI type and > the > >>>>>>>> Dynamic > >>>>>>>>>>>>> type: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> iSCSI type: The admin has to go in and create a Primary > >> Storage > >>>>>>>> based > >>>>>>>>>>>>> on > >>>>>>>>>>>>>> the iSCSI type. At this point in time, the iSCSI volume mu= st > >>>>> exist > >>>>>>>> on > >>>>>>>>>>>>> the > >>>>>>>>>>>>>> storage system (it is pre-allocated). Future CloudStack > >> volumes > >>>>> are > >>>>>>>>>>>>> created > >>>>>>>>>>>>>> as VDIs on the SR that was created behind the scenes. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dynamic type: The admin has to go in and create Primary > >> Storage > >>>>>>>> based > >>>>>>>>>>>>> on a > >>>>>>>>>>>>>> plug-in that will create and delete volumes on its storage > >>>>> system > >>>>>>>>>>>>>> dynamically (as is enabled via the storage framework). Whe= n > a > >>>>> user > >>>>>>>>>>>>> wants to > >>>>>>>>>>>>>> attach a CloudStack volume that was created, the framework > >> tells > >>>>>>>> the > >>>>>>>>>>>>>> plug-in to create a new volume. After this is done, the > attach > >>>>>>>> logic > >>>>>>>>>>>>> for > >>>>>>>>>>>>>> the hypervisor in question is called. No hypervisor data > >>>>> structure > >>>>>>>>>>>>> exists > >>>>>>>>>>>>>> at this point because the volume was just created. The > >>>>> hypervisor > >>>>>>>> data > >>>>>>>>>>>>>> structure must be created. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:21 PM, Mike Tutkowski < > >>>>>>>>>>>>> mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> These are new terms, so I should probably have defined th= em > >> up > >>>>>>>> front > >>>>>>>>>>>>> for > >>>>>>>>>>>>>>> you. :) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Static storage: Storage that is pre-allocated (ex. an adm= in > >>>>>>>> creates a > >>>>>>>>>>>>>>> volume on a SAN), then a hypervisor data structure is > created > >>>>> to > >>>>>>>>>>>>> consume > >>>>>>>>>>>>>>> the storage (ex. XenServer SR), then that hypervisor data > >>>>>>>> structure > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>> consumed by CloudStack. Disks (VDI) are later placed on > this > >>>>>>>>>>>>> hypervisor > >>>>>>>>>>>>>>> data structure as needed. In these cases, the attach logi= c > >>>>> assumes > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>> hypervisor data structure is already in place and simply > >>>>> attaches > >>>>>>>>>>>>> the VDI > >>>>>>>>>>>>>>> on the hypervisor data structure to the VM in question. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Dynamic storage: Storage that is not pre-allocated. Inste= ad > >> of > >>>>>>>>>>>>>>> pre-existent storage, this could be a SAN (not a volume o= n > a > >>>>> SAN, > >>>>>>>>>>>>> but the > >>>>>>>>>>>>>>> SAN itself). The hypervisor data structure must be create= d > >>>>> when an > >>>>>>>>>>>>> attach > >>>>>>>>>>>>>>> volume is performed because these types of volumes have n= ot > >>>>> been > >>>>>>>>>>>>> pre-hooked > >>>>>>>>>>>>>>> up to such a hypervisor data structure by an admin. Once > the > >>>>>>>> attach > >>>>>>>>>>>>> logic > >>>>>>>>>>>>>>> creates, say, an SR on XenServer for this volume, it > attaches > >>>>> the > >>>>>>>>>>>>> one and > >>>>>>>>>>>>>>> only VDI within the SR to the VM in question. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:13 PM, John Burwell < > >>>>> jburwell@basho.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Mike, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The current implementation of the Dynamic type attach > >> behavior > >>>>>>>>>>>>> works in > >>>>>>>>>>>>>>>> terms of Xen ISCSI which why I ask about the difference. > >>>>> Another > >>>>>>>>>>>>> way to > >>>>>>>>>>>>>>>> ask the question -- what is the definition of a Dynamic > >>>>> storage > >>>>>>>>>>>>> pool type? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Jun 3, 2013, at 5:10 PM, Mike Tutkowski < > >>>>>>>>>>>>> mike.tutkowski@solidfire.com> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> As far as I know, the iSCSI type is uniquely used by > >>>>> XenServer > >>>>>>>>>>>>> when you > >>>>>>>>>>>>>>>>> want to set up Primary Storage that is directly based o= n > an > >>>>>>>> iSCSI > >>>>>>>>>>>>>>>> target. > >>>>>>>>>>>>>>>>> This allows you to skip the step of going to the > hypervisor > >>>>> and > >>>>>>>>>>>>>>>> creating a > >>>>>>>>>>>>>>>>> storage repository based on that iSCSI target as > CloudStack > >>>>> does > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> part > >>>>>>>>>>>>>>>>> for you. I think this is only supported for XenServer. > For > >>>>> all > >>>>>>>>>>>>> other > >>>>>>>>>>>>>>>>> hypervisors, you must first go to the hypervisor and > >> perform > >>>>>>>> this > >>>>>>>>>>>>> step > >>>>>>>>>>>>>>>>> manually. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I don't really know what RBD is. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 2:13 PM, John Burwell < > >>>>>>>> jburwell@basho.com> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Mike, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Reading through the code, what is the difference betwe= en > >> the > >>>>>>>>>>>>> ISCSI and > >>>>>>>>>>>>>>>>>> Dynamic types? Why isn't RBD considered Dynamic? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 3:46 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> This new type of storage is defined in the > >>>>>>>>>>>>> Storage.StoragePoolType > >>>>>>>>>>>>>>>> class > >>>>>>>>>>>>>>>>>>> (called Dynamic): > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> public static enum StoragePoolType { > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Filesystem(false), // local directory > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> NetworkFilesystem(true), // NFS or CIFS > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> IscsiLUN(true), // shared LUN, with a clusterfs > overlay > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Iscsi(true), // for e.g., ZFS Comstar > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ISO(false), // for iso image > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> LVM(false), // XenServer local LVM SR > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> CLVM(true), > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> RBD(true), > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> SharedMountPoint(true), > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> VMFS(true), // VMware VMFS storage > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> PreSetup(true), // for XenServer, Storage Pool is se= t > >> up > >>>>>>>> by > >>>>>>>>>>>>>>>>>>> customers. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> EXT(false), // XenServer local EXT SR > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> OCFS2(true), > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Dynamic(true); // dynamic, zone-wide storage (ex. > >>>>>>>> SolidFire) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> boolean shared; > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> StoragePoolType(boolean shared) { > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> this.shared =3D shared; > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> public boolean isShared() { > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> return shared; > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:41 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> For example, let's say another storage company wants > to > >>>>>>>>>>>>> implement a > >>>>>>>>>>>>>>>>>>>> plug-in to leverage its Quality of Service feature. = It > >>>>> would > >>>>>>>> be > >>>>>>>>>>>>>>>> dynamic, > >>>>>>>>>>>>>>>>>>>> zone-wide storage, as well. They would need only > >>>>> implement a > >>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>> plug-in as I've made the necessary changes to the > >>>>>>>>>>>>> hypervisor-attach > >>>>>>>>>>>>>>>>>> logic > >>>>>>>>>>>>>>>>>>>> to support their plug-in. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:39 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Oh, sorry to imply the XenServer code is SolidFire > >>>>> specific. > >>>>>>>>>>>>> It is > >>>>>>>>>>>>>>>> not. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> The XenServer attach logic is now aware of dynamic, > >>>>>>>> zone-wide > >>>>>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>> (and SolidFire is an implementation of this kind of > >>>>>>>> storage). > >>>>>>>>>>>>> This > >>>>>>>>>>>>>>>>>> kind of > >>>>>>>>>>>>>>>>>>>>> storage is new to 4.2 with Edison's storage framewo= rk > >>>>>>>> changes. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Edison created a new framework that supported the > >>>>> creation > >>>>>>>> and > >>>>>>>>>>>>>>>> deletion > >>>>>>>>>>>>>>>>>>>>> of volumes dynamically. However, when I visited wit= h > >> him > >>>>> in > >>>>>>>>>>>>> Portland > >>>>>>>>>>>>>>>>>> back > >>>>>>>>>>>>>>>>>>>>> in April, we realized that it was not complete. We > >>>>> realized > >>>>>>>>>>>>> there > >>>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>>>>>> nothing CloudStack could do with these volumes unle= ss > >> the > >>>>>>>>>>>>> attach > >>>>>>>>>>>>>>>> logic > >>>>>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>>>>>> changed to recognize this new type of storage and > >> create > >>>>> the > >>>>>>>>>>>>>>>>>> appropriate > >>>>>>>>>>>>>>>>>>>>> hypervisor data structure. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:28 PM, John Burwell < > >>>>>>>>>>>>> jburwell@basho.com> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Mike, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> It is generally odd to me that any operation in th= e > >>>>> Storage > >>>>>>>>>>>>> layer > >>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> understand or care about details. I expect to see > the > >>>>>>>> Storage > >>>>>>>>>>>>>>>>>> services > >>>>>>>>>>>>>>>>>>>>>> expose a set of operations that can be > composed/driven > >>>>> by > >>>>>>>> the > >>>>>>>>>>>>>>>>>> Hypervisor > >>>>>>>>>>>>>>>>>>>>>> implementations to allocate space/create structure= s > >> per > >>>>>>>> their > >>>>>>>>>>>>>>>> needs. > >>>>>>>>>>>>>>>>>> If > >>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>> don't invert this dependency, we are going to end > >> with a > >>>>>>>>>>>>> massive > >>>>>>>>>>>>>>>>>> n-to-n > >>>>>>>>>>>>>>>>>>>>>> problem that will make the system increasingly > >>>>> difficult to > >>>>>>>>>>>>>>>> maintain > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> enhance. Am I understanding that the Xen specific > >>>>>>>> SolidFire > >>>>>>>>>>>>> code > >>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> located in the CitrixResourceBase class? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 3:12 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> To delve into this in a bit more detail: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Prior to 4.2 and aside from one setup method for > >>>>>>>> XenServer, > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> admin > >>>>>>>>>>>>>>>>>>>>>> had > >>>>>>>>>>>>>>>>>>>>>>> to first create a volume on the storage system, > then > >> go > >>>>>>>> into > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> hypervisor > >>>>>>>>>>>>>>>>>>>>>>> to set up a data structure to make use of the > volume > >>>>> (ex. > >>>>>>>> a > >>>>>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>>>> repository on XenServer or a datastore on ESX(i))= . > >> VMs > >>>>> and > >>>>>>>>>>>>> data > >>>>>>>>>>>>>>>> disks > >>>>>>>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>>>>>>> shared this storage system's volume. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> With Edison's new storage framework, storage need > no > >>>>>>>> longer > >>>>>>>>>>>>> be so > >>>>>>>>>>>>>>>>>>>>>> static > >>>>>>>>>>>>>>>>>>>>>>> and you can easily create a 1:1 relationship > between > >> a > >>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>> system's > >>>>>>>>>>>>>>>>>>>>>>> volume and the VM's data disk (necessary for > storage > >>>>>>>> Quality > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>> Service). > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> You can now write a plug-in that is called to > >>>>> dynamically > >>>>>>>>>>>>> create > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> delete > >>>>>>>>>>>>>>>>>>>>>>> volumes as needed. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> The problem that the storage framework did not > >> address > >>>>> is > >>>>>>>> in > >>>>>>>>>>>>>>>> creating > >>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>> deleting the hypervisor-specific data structure > when > >>>>>>>>>>>>> performing an > >>>>>>>>>>>>>>>>>>>>>>> attach/detach. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> That being the case, I've been enhancing it to do > so. > >>>>> I've > >>>>>>>>>>>>> got > >>>>>>>>>>>>>>>>>>>>>> XenServer > >>>>>>>>>>>>>>>>>>>>>>> worked out and submitted. I've got ESX(i) in my > >> sandbox > >>>>>>>> and > >>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>> submit > >>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>> if we extend the 4.2 freeze date. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Does that help a bit? :) > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 1:03 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi John, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> The storage plug-in - by itself - is hypervisor > >>>>> agnostic. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> The issue is with the volume-attach logic (in th= e > >>>>> agent > >>>>>>>>>>>>> code). > >>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>>>>> framework calls into the plug-in to have it > create a > >>>>>>>> volume > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>> needed, > >>>>>>>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>>>>> when the time comes to attach the volume to a > >>>>> hypervisor, > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> attach > >>>>>>>>>>>>>>>>>>>>>>> logic > >>>>>>>>>>>>>>>>>>>>>>>> has to be smart enough to recognize it's being > >>>>> invoked on > >>>>>>>>>>>>>>>> zone-wide > >>>>>>>>>>>>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>>>>> (where the volume has just been created) and > create, > >>>>>>>> say, a > >>>>>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>>>>> repository (for XenServer) or a datastore (for > >>>>> VMware) to > >>>>>>>>>>>>> make > >>>>>>>>>>>>>>>> use > >>>>>>>>>>>>>>>>>>>>>> of the > >>>>>>>>>>>>>>>>>>>>>>>> volume that was just created. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I've been spending most of my time recently maki= ng > >> the > >>>>>>>>>>>>> attach > >>>>>>>>>>>>>>>> logic > >>>>>>>>>>>>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>>>>>>>> in the agent code. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Does that clear it up? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 12:48 PM, John Burwell < > >>>>>>>>>>>>>>>> jburwell@basho.com> > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Mike, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why the the storage driver is > >>>>> hypervisor > >>>>>>>>>>>>>>>> specific? > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 1:21 PM, Mike Tutkowski < > >>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Yes, ultimately I would like to support all > >>>>> hypervisors > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>> CloudStack > >>>>>>>>>>>>>>>>>>>>>>>>>> supports. I think I'm just out of time for 4.2 > to > >>>>> get > >>>>>>>> KVM > >>>>>>>>>>>>> in. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Right now this plug-in supports XenServer. > >>>>> Depending on > >>>>>>>>>>>>> what > >>>>>>>>>>>>>>>> we do > >>>>>>>>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>>>> regards to 4.2 feature freeze, I have it worki= ng > >> for > >>>>>>>>>>>>> VMware in > >>>>>>>>>>>>>>>> my > >>>>>>>>>>>>>>>>>>>>>>>>> sandbox, > >>>>>>>>>>>>>>>>>>>>>>>>>> as well. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Also, just to be clear, this is all in regards > to > >>>>> Disk > >>>>>>>>>>>>>>>> Offerings. > >>>>>>>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>>>>>> plan to > >>>>>>>>>>>>>>>>>>>>>>>>>> support Compute Offerings post 4.2. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 11:14 AM, Kelcey Jamiso= n > >>>>> Damage > >>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> kelcey@bbits.ca > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any plan on supporting KVM in the > patch > >>>>> cycle > >>>>>>>>>>>>> post > >>>>>>>>>>>>>>>> 4.2? > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Mike Tutkowski" < > >>>>> mike.tutkowski@solidfire.com> > >>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@cloudstack.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Monday, June 3, 2013 10:12:32 AM > >>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [MERGE] disk_io_throttling to > MASTER > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I agree on merging Wei's feature first, then > >> mine. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> If his feature is for KVM only, then it is a > non > >>>>> issue > >>>>>>>>>>>>> as I > >>>>>>>>>>>>>>>> don't > >>>>>>>>>>>>>>>>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>>>>>>> KVM in 4.2. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 3, 2013 at 8:55 AM, Wei ZHOU < > >>>>>>>>>>>>>>>> ustcweizhou@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> For the billing, as no one works on billing > now, > >>>>>>>> users > >>>>>>>>>>>>> need > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>> calculate > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the billing by themselves. They can get the > >>>>>>>>>>>>> service_offering > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_offering of a VMs and volumes for > >>>>> calculation. > >>>>>>>> Of > >>>>>>>>>>>>> course > >>>>>>>>>>>>>>>>>>>>>> it is > >>>>>>>>>>>>>>>>>>>>>>>>>>> better > >>>>>>>>>>>>>>>>>>>>>>>>>>>> to tell user the exact limitation value of > >>>>> individual > >>>>>>>>>>>>> volume, > >>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>> network > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rate limitation for nics as well. I can work > on > >> it > >>>>>>>>>>>>> later. Do > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>>>>> think it > >>>>>>>>>>>>>>>>>>>>>>>>>>>> is a part of I/O throttling? > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry my misunstand the second the question. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Agree with what you said about the two > features. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 3, 2013, at 2:13 AM, Wei ZHOU < > >>>>>>>>>>>>> ustcweizhou@gmail.com > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John, Mike > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope Mike's aswer helps you. I am trying > to > >>>>>>>> adding > >>>>>>>>>>>>> more. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) I think billing should depend on IO > >>>>> statistics > >>>>>>>>>>>>> rather > >>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>>>>>>>> IOPS > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation. Please review disk_io_stat if > you > >>>>> have > >>>>>>>>>>>>> time. > >>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_stat > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get the IO statistics including bytes/iops > >>>>>>>> read/write > >>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>>> individual > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virtual machine. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Going by the AWS model, customers are bille= d > >> more > >>>>>>>> for > >>>>>>>>>>>>>>>> volumes > >>>>>>>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioned IOPS, as well as, for those > >>>>> operations ( > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://aws.amazon.com/ebs/). I would > imagine > >>>>> our > >>>>>>>>>>>>> users > >>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> option to employ similar cost models. Coul= d > an > >>>>>>>>>>>>> operator > >>>>>>>>>>>>>>>>>>>>>> implement > >>>>>>>>>>>>>>>>>>>>>>>>>>> such a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> billing model in the current patch? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) Do you mean IOPS runtime change? KVM > >>>>> supports > >>>>>>>>>>>>> setting > >>>>>>>>>>>>>>>>>>>>>> IOPS/BPS > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> limitation for a running virtual machine > >> through > >>>>>>>>>>>>> command > >>>>>>>>>>>>>>>> line. > >>>>>>>>>>>>>>>>>>>>>>>>>>> However, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CloudStack does not support changing the > >>>>> parameters > >>>>>>>>>>>>> of a > >>>>>>>>>>>>>>>>>>>>>> created > >>>>>>>>>>>>>>>>>>>>>>>>>>>> offering > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (computer offering or disk offering). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I meant at the Java interface level. I > >> apologize > >>>>>>>> for > >>>>>>>>>>>>> being > >>>>>>>>>>>>>>>>>>>>>>> unclear. > >>>>>>>>>>>>>>>>>>>>>>>>>>> Can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we more generalize allocation algorithms > with a > >>>>> set > >>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>> interfaces > >>>>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> describe the service guarantees provided by= a > >>>>>>>> resource? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) It is a good question. Maybe it is > better > >> to > >>>>>>>>>>>>> commit > >>>>>>>>>>>>>>>> Mike's > >>>>>>>>>>>>>>>>>>>>>>> patch > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> after > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling as Mike needs to consid= er > >> the > >>>>>>>>>>>>>>>> limitation in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type, I think. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will expand on my thoughts in a later > >> response > >>>>> to > >>>>>>>>>>>>> Mike > >>>>>>>>>>>>>>>>>>>>>> regarding > >>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> touch points between these two features. I > >> think > >>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will need to be merged before SolidFire, bu= t > I > >>>>> think > >>>>>>>>>>>>> we need > >>>>>>>>>>>>>>>>>>>>>> closer > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> coordination between the branches (possibly > >> have > >>>>>>>> have > >>>>>>>>>>>>>>>> solidfire > >>>>>>>>>>>>>>>>>>>>>>> track > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_io_throttling) to coordinate on this > >> issue. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/3 John Burwell > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mike, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The things I want to understand are the > >>>>> following: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Is there value in capturing IOPS > policies > >> be > >>>>>>>>>>>>> captured > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>> common > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data model (e.g. for billing/usage > purposes, > >>>>>>>>>>>>> expressing > >>>>>>>>>>>>>>>>>>>>>>> offerings). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Should there be a common interface mod= el > >> for > >>>>>>>>>>>>> reasoning > >>>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>>>>>>> IOP > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provisioning at runtime? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) How are conflicting provisioned IOPS > >>>>>>>>>>>>> configurations > >>>>>>>>>>>>>>>>>>>>>> between > >>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hypervisor and storage device reconciled? > In > >>>>>>>>>>>>> particular, > >>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>> scenario > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> where > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is lead to believe (and billed) for more > IOPS > >>>>>>>>>>>>> configured > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> a VM > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage device has been configured to > >> deliver. > >>>>>>>>>>>>> Another > >>>>>>>>>>>>>>>>>>>>>> scenario > >>>>>>>>>>>>>>>>>>>>>>>>>>>> could a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent configuration between a VM and= a > >>>>>>>> storage > >>>>>>>>>>>>>>>> device at > >>>>>>>>>>>>>>>>>>>>>>>>>>> creation > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time, but a later modification to storage > >>>>> device > >>>>>>>>>>>>>>>> introduces > >>>>>>>>>>>>>>>>>>>>>>> logical > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistency. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 2, 2013, at 8:38 PM, Mike Tutkowsk= i > < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi John, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe Wei's feature deals with > >> controlling > >>>>> the > >>>>>>>>>>>>> max > >>>>>>>>>>>>>>>>>>>>>> number of > >>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the hypervisor side. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My feature is focused on controlling IOPS > >> from > >>>>> the > >>>>>>>>>>>>> storage > >>>>>>>>>>>>>>>>>>>>>> system > >>>>>>>>>>>>>>>>>>>>>>>>>>>> side. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope that helps. :) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Jun 2, 2013 at 6:35 PM, John > Burwell > >> < > >>>>>>>>>>>>>>>>>>>>>> jburwell@basho.com > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My opinion is that no features should be > >>>>> merged > >>>>>>>>>>>>> until all > >>>>>>>>>>>>>>>>>>>>>>>>>>> functional > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues have been resolved and it is read= y > to > >>>>> turn > >>>>>>>>>>>>> over to > >>>>>>>>>>>>>>>>>>>>>> test. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Until > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total Ops vs discrete read/write ops iss= ue > >> is > >>>>>>>>>>>>> addressed > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>> re-reviewed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, I don't think this criteria has be= en > >>>>>>>>>>>>> satisfied. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, how does this work > >> intersect/compliment > >>>>> the > >>>>>>>>>>>>>>>> SolidFire > >>>>>>>>>>>>>>>>>>>>>>> patch > >>>>>>>>>>>>>>>>>>>>>>>>> ( > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://reviews.apache.org/r/11479/)? > As I > >>>>>>>>>>>>> understand > >>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also involves provisioned IOPS. I would > >> like > >>>>> to > >>>>>>>>>>>>> ensure > >>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>> don't > >>>>>>>>>>>>>>>>>>>>>>>>>>>> have a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scenario where provisioned IOPS in KVM a= nd > >>>>>>>>>>>>> SolidFire are > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incompatible. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jun 1, 2013, at 6:47 AM, Wei ZHOU < > >>>>>>>>>>>>>>>> ustcweizhou@gmail.com > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sure. I will change it next week. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/6/1 Wido den Hollander < > wido@widodh.nl > >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 06/01/2013 08:24 AM, Wei ZHOU wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exactly. I have pushed the features into > >>>>> master. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If anyone object thems for technical > reason > >>>>> till > >>>>>>>>>>>>> Monday, > >>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>>>>>>>> revert > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the sake of clarity I just want to > >> mention > >>>>>>>> again > >>>>>>>>>>>>>>>> that we > >>>>>>>>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> change > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the total IOps to R/W IOps asap so that = we > >>>>> never > >>>>>>>>>>>>> release > >>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>> version > >>>>>>>>>>>>>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only total IOps. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You laid the groundwork for the I/O > >> throttling > >>>>>>>> and > >>>>>>>>>>>>> that's > >>>>>>>>>>>>>>>>>>>>>> great! > >>>>>>>>>>>>>>>>>>>>>>>>> We > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however prevent that we create legacy fr= om > >> day > >>>>>>>> #1. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander < > >> wido@widodh.nl> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:59 PM, John Burwell > wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 -- this enhancement must to discretel= y > >>>>> support > >>>>>>>>>>>>> read > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> write > >>>>>>>>>>>>>>>>>>>>>>>>>>>> IOPS. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how it could be fixed later > >> because > >>>>> I > >>>>>>>>>>>>> don't see > >>>>>>>>>>>>>>>>>>>>>> how we > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correctly > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> split total IOPS into read and write. > >>>>>>>> Therefore, we > >>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>> stuck > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total unless/until we decided to break > >>>>> backwards > >>>>>>>>>>>>>>>>>>>>>> compatibility. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What Wei meant was merging it into maste= r > >> now > >>>>> so > >>>>>>>>>>>>> that it > >>>>>>>>>>>>>>>>>>>>>> will go > >>>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2 branch and add Read / Write IOps > before > >>>>> the > >>>>>>>> 4.2 > >>>>>>>>>>>>>>>> release > >>>>>>>>>>>>>>>>>>>>>> so > >>>>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.2 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be released with Read and Write > instead > >>>>> of > >>>>>>>>>>>>> Total > >>>>>>>>>>>>>>>> IOps. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is to make the May 31st feature > freeze > >>>>> date. > >>>>>>>>>>>>> But if > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> window > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> moves > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (see other threads) then it won't be > >> necessary > >>>>>>>> to do > >>>>>>>>>>>>>>>> that. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I also completely agree that there is no > >>>>>>>> association > >>>>>>>>>>>>>>>> between > >>>>>>>>>>>>>>>>>>>>>>>>>>> network > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk I/O. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -John > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On May 31, 2013, at 9:51 AM, Wido den > >>>>> Hollander < > >>>>>>>>>>>>>>>>>>>>>> wido@widodh.nl > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/31/2013 03:13 PM, Wei ZHOU wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wido, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. Good question. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I thought about at the beginning. > Finally I > >>>>>>>>>>>>> decided to > >>>>>>>>>>>>>>>>>>>>>> ignore > >>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difference of read and write mainly > because > >>>>> the > >>>>>>>>>>>>> network > >>>>>>>>>>>>>>>>>>>>>>> throttling > >>>>>>>>>>>>>>>>>>>>>>>>>>>> did > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> care the difference of sent and received > >>>>> bytes as > >>>>>>>>>>>>> well. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That reasoning seems odd. Networking and > >> disk > >>>>> I/O > >>>>>>>>>>>>>>>> completely > >>>>>>>>>>>>>>>>>>>>>>>>>>>> different. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Disk I/O is much more expensive in most > >>>>>>>> situations > >>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>>>>>> network > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> bandwith. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implementing it will be some copy-paste > >> work. > >>>>> It > >>>>>>>>>>>>> could be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> few days. For the deadline of feature > >> freeze, > >>>>> I > >>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>> implement > >>>>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that , if needed. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It think it's a feature we can't miss. B= ut > >> if > >>>>> it > >>>>>>>>>>>>> goes > >>>>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> 4.2 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> window we have to make sure we don't > release > >>>>> with > >>>>>>>>>>>>> only > >>>>>>>>>>>>>>>> total > >>>>>>>>>>>>>>>>>>>>>>> IOps > >>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fix > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it in 4.3, that will confuse users. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/5/31 Wido den Hollander < > >> wido@widodh.nl> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Wei, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 05/30/2013 06:03 PM, Wei ZHOU wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to merge disk_io_throttling > >>>>> branch > >>>>>>>> into > >>>>>>>>>>>>>>>> master. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If nobody object, I will merge into mast= er > >> in > >>>>> 48 > >>>>>>>>>>>>> hours. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The purpose is : > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Virtual machines are running on the same > >>>>> storage > >>>>>>>>>>>>> device > >>>>>>>>>>>>>>>>>>>>>> (local > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage or > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> share strage). Because of the rate > >> limitation > >>>>> of > >>>>>>>>>>>>> device > >>>>>>>>>>>>>>>>>>>>>> (such as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> iops), if > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one VM has large disk operation, it may > >> affect > >>>>>>>> the > >>>>>>>>>>>>> disk > >>>>>>>>>>>>>>>>>>>>>>>>> performance > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other VMs running on the same storage > >> device. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is neccesary to set the maximum rate > and > >>>>> limit > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> disk > >>>>>>>>>>>>>>>>>>>>>> I/O > >>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VMs. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking at the code I see you make no > >>>>> difference > >>>>>>>>>>>>> between > >>>>>>>>>>>>>>>>>>>>>> Read > >>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Qemu and libvirt support setting both a > >>>>> different > >>>>>>>>>>>>> rate > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> Read > >>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps which could benefit a lot of users. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's also strange, in the polling side y= ou > >>>>>>>> collect > >>>>>>>>>>>>> both > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> Read > >>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps, but on the throttling side you onl= y > go > >>>>> for > >>>>>>>> a > >>>>>>>>>>>>> global > >>>>>>>>>>>>>>>>>>>>>> value. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Write IOps are usually much more expensi= ve > >>>>> then > >>>>>>>> Read > >>>>>>>>>>>>>>>> IOps, > >>>>>>>>>>>>>>>>>>>>>> so it > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like a valid use-case where that an admi= n > >>>>> would > >>>>>>>> set > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>>> lower > >>>>>>>>>>>>>>>>>>>>>>> value > >>>>>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IOps vs Read IOps. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since this only supports KVM at this > point I > >>>>>>>> think > >>>>>>>>>>>>> it > >>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value to at least have the mechanism in > >> place > >>>>> to > >>>>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>> both, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this later would be a lot of work. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If a hypervisor doesn't support setting > >>>>> different > >>>>>>>>>>>>> values > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>> read > >>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> write you can always sum both up and set > >> that > >>>>> as > >>>>>>>> the > >>>>>>>>>>>>>>>> total > >>>>>>>>>>>>>>>>>>>>>>> limit. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you explain why you implemented it > this > >>>>> way? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wido > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The feature includes: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the maximum rate of VMs (in > >>>>>>>> disk_offering, > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> global > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) change the maximum rate of VMs > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) limit the disk rate (total bps and > iops) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> JIRA ticket: > https://issues.apache.org/**** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192 >>>>>>>>>>>>>>>>>>>>>> issues.apache.org/**** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jira/browse/CLOUDSTACK-1192< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1192< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1192> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1192< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1192> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FS (I will update later) : > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>> > https://cwiki.apache.org/******confluence/display/CLOUDSTACK/****** > >>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/** > >>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > https://cwiki.apache.org/**confluence/display/CLOUDSTACK/** > >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling >>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/**** > >>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> display/CLOUDSTACK/VM+Disk+IO+****Throttling< > >>>>>>>>>>>>>>>> https://cwiki. > >>>>>>>>>>>>>>>>>>>>>> ** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>> > apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling > >>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttl= ing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Merge check list :- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Did you check the branch's RAT executi= on > >>>>>>>> success? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * Are there new dependencies introduced? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What automated testing (unit and > >>>>> integration) > >>>>>>>> is > >>>>>>>>>>>>>>>> included > >>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Unit tests are added. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * What testing has been done to check fo= r > >>>>>>>> potential > >>>>>>>>>>>>>>>>>>>>>> regressions? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) set the bytes rate and IOPS rate on > >>>>>>>> CloudStack > >>>>>>>>>>>>> UI. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (2) VM operations, including > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deploy, stop, start, reboot, destroy, > >> expunge. > >>>>>>>>>>>>> migrate, > >>>>>>>>>>>>>>>>>>>>>> restore > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (3) Volume operations, including > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Attach, Detach > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To review the code, you can try > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> git diff > >>>>>>>>>>>>> c30057635d04a2396f84c588127d7e******be42e503a7 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> f2e5591b710d04cc86815044f5823e******73a4a58944 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wei > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>> > https://cwiki.apache.org/******confluence/display/CLOUDSTACK/****** > >>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >> https://cwiki.apache.org/****confluence/display/CLOUDSTACK/****> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >> https://cwiki.apache.org/****confluence/display/**CLOUDSTACK/** > >>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > https://cwiki.apache.org/**confluence/display/CLOUDSTACK/** > >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VM+Disk+IO+Throttling >>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org/confluence/**** > >>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://cwiki.apache.org/confluence/**> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> display/CLOUDSTACK/VM+Disk+IO+****Throttling< > >>>>>>>>>>>>>>>> https://cwiki. > >>>>>>>>>>>>>>>>>>>>>> ** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>> > apache.org/confluence/display/**CLOUDSTACK/VM+Disk+IO+**Throttling > >>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Disk+IO+Throttl= ing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] refs/heads/disk_io_throttling > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > https://issues.apache.org/******jira/browse/CLOUDSTACK-1301 > >>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-1301 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> issues.apache.org/****jira/browse/CLOUDSTACK-1301< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> issues.apache.org/**jira/**browse/CLOUDSTACK-1301< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-1301> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-1301< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1301> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> issues.apache.org/****jira/**browse/CLOUDSTACK-207= 1 > < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> http://issues.apache.org/**jira/**browse/CLOUDSTACK-2071 > >>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> http://issues.apache.org/jira/**browse/CLOUDSTACK-2071> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <** > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://issues.apache.org/****jira/browse/CLOUDSTACK-207= 1 > < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/**jira/browse/CLOUDSTACK-2071> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> issues.apache.org/jira/**browse/CLOUDSTACK-2071< > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-2071> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (**CLOUDSTACK-1301 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - VM Disk I/O Throttling) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire > Inc.* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloud< > >>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>>>>>>>>>>>> cloud< > >>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>>>>>>>>>>> cloud< > >>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dp= lay > > > >>>>>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>>>>>>>> cloud< > >>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>>>> cloud< > http://solidfire.com/solution/overview/?video=3Dplay > >>> > >>>>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>>>>> cloud > > >>>>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>>>> Advancing the way the world uses the > >>>>>>>>>>>>>> cloud > >>>>>>>>>>>>>> *=99* > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>>> *=99* > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>>> o: 303.746.7302 > >>>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>>> *=99* > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> *Mike Tutkowski* > >>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>>> o: 303.746.7302 > >>>>>>>>>> Advancing the way the world uses the cloud< > >>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>>>>> *=99* > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> *Mike Tutkowski* > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>>>> o: 303.746.7302 > >>>>>>>>> Advancing the way the world uses the > >>>>>>>>> cloud > >>>>>>>>> *=99* > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> *Mike Tutkowski* > >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>>> e: mike.tutkowski@solidfire.com > >>>>>>> o: 303.746.7302 > >>>>>>> Advancing the way the world uses the cloud< > >>>>> http://solidfire.com/solution/overview/?video=3Dplay> > >>>>>>> *=99* > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> *Mike Tutkowski* > >>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>>>> e: mike.tutkowski@solidfire.com > >>>>>> o: 303.746.7302 > >>>>>> Advancing the way the world uses the > >>>>>> cloud > >>>>>> *=99* > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> *Mike Tutkowski* > >>>> *Senior CloudStack Developer, SolidFire Inc.* > >>>> e: mike.tutkowski@solidfire.com > >>>> o: 303.746.7302 > >>>> Advancing the way the world uses the cloud< > >> http://solidfire.com/solution/overview/?video=3Dplay> > >>>> *=99* > >>>> > >>> > >>> > >>> > >>> -- > >>> *Mike Tutkowski* > >>> *Senior CloudStack Developer, SolidFire Inc.* > >>> e: mike.tutkowski@solidfire.com > >>> o: 303.746.7302 > >>> Advancing the way the world uses the > >>> cloud > >>> *=99* > >> > >> > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkowski@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud > > *=99* > > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --047d7b5d300c5a9fac04de5ac3f4--