Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 489E21069E for ; Wed, 12 Jun 2013 22:31:27 +0000 (UTC) Received: (qmail 87261 invoked by uid 500); 12 Jun 2013 22:31:26 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 87223 invoked by uid 500); 12 Jun 2013 22:31:26 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 87215 invoked by uid 99); 12 Jun 2013 22:31:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jun 2013 22:31:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of mike.tutkowski@solidfire.com designates 209.85.214.178 as permitted sender) Received: from [209.85.214.178] (HELO mail-ob0-f178.google.com) (209.85.214.178) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jun 2013 22:31:21 +0000 Received: by mail-ob0-f178.google.com with SMTP id fb19so13971812obc.37 for ; Wed, 12 Jun 2013 15:31:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=yANhxL6L0CIHbqsA3borXu7/58y0s9/VxTEp0+U3hN0=; b=daw8bzvKDKNlvOpLEcc4nHitoPy9AMhiE4kB5VGw7zA1rbeLTi3knqxjvlwldtF/Y2 vbR3EX9TvCcHHiEeWFIs2uH9bVd16zVKpPQy54FhcVmY4wLDX1Os7ovH3Muv6QlBJiR5 ze90YT2icmB6fxVmRmiBA9clXf5T89Sw/bCixJlAFoWRjCAxuwAkQ1isjSwwWb6OcqiS rmQHdBuI6ORm1MB+HhlSJdnKZ6Wk9zfJorbb5e5dUsynO5RopMlCYrFPiZ9SY3shoF5u lNBA9iiDLm+yV4tF9hWLMGDdJMJzTjfOb5udECUtWeowiGkHC0PPHGHOLqxPlKxgDE1E 0VJQ== MIME-Version: 1.0 X-Received: by 10.182.213.10 with SMTP id no10mr16676723obc.76.1371076260373; Wed, 12 Jun 2013 15:31:00 -0700 (PDT) Received: by 10.182.10.66 with HTTP; Wed, 12 Jun 2013 15:31:00 -0700 (PDT) In-Reply-To: References: <12535A06-F161-4739-855F-99B4A001EFAF@basho.com> <77B337AF224FD84CBF8401947098DD87039664@SJCPEX01CL01.citrite.net> <77B337AF224FD84CBF8401947098DD870396BD@SJCPEX01CL01.citrite.net> <77B337AF224FD84CBF8401947098DD8703977F@SJCPEX01CL01.citrite.net> <77B337AF224FD84CBF8401947098DD8704A7E8@SJCPEX01CL01.citrite.net> <4BC4E13F-B99C-44ED-B6FA-81CB9373A80F@basho.com> <77B337AF224FD84CBF8401947098DD8704BAB8@SJCPEX01CL01.citrite.net> <1130E63E-C93A-45C4-A96D-F375E9BB1C4F@basho.com> <77B337AF224FD84CBF8401947098DD8704BB28@SJCPEX01CL01.citrite.net> <018F234D-0552-4B0D-BF73-AB25876D960B@basho.com> <56A0CF3D-4BCB-4B78-8E1D-8DE3003B6948@basho.com> <4CCCF606-9094-475B-BF85-77AD2D95F901@basho.com> <59871011-033B-41C1-87D4-AD84B2B01752@basho.com> <3954AAAC-9C13-4621-89DB-E3B1F4C3502D@basho.com> Date: Wed, 12 Jun 2013 16:31:00 -0600 Message-ID: Subject: Re: [MERGE] disk_io_throttling to MASTER From: Mike Tutkowski To: John Burwell Cc: "dev@cloudstack.apache.org" , Wei Zhou Content-Type: multipart/alternative; boundary=001a11c2deaa75abbd04defc9181 X-Gm-Message-State: ALoCoQkl8QntZol3IVmQQVTI+rbMEBWoKwmrJ2UbO9u+V7yWoOQvCiJYQdmLYCFO/TsXJhDzhlyD X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2deaa75abbd04defc9181 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable We'd also have to recognize that if the Min value is filled in and Storage QoS is not available that Hypervisor QoS (rate limiting here) cannot satisfy that constraint. On Wed, Jun 12, 2013 at 4:26 PM, Mike Tutkowski < mike.tutkowski@solidfire.com> wrote: > Yeah, this is interesting. > > We'll have to wait and see what Wei's thoughts are on this. > > > On Wed, Jun 12, 2013 at 4:17 PM, John Burwell wrote: > >> Mike, >> >> Yes. To your point, the appropriate logic would be to check the Volume >> allocated by the StorageAllocator. If it doesn't support a QoS, then th= e >> VM allocator would attempt to fulfill the QoS through the hypervisor. >> Another question would be -- what would be in the behavior in the event >> that no resources are available to fulfill the QoS? We could outright f= ail >> or proceed with some kind of warning -- sounds like a good place for a >> global setting to configure the policy. >> >> The other question we haven't answered are usage records. By pushing th= e >> QoS concept out to allocation and making it a more general concept, it >> could make usage record capture more straightforward. >> >> Thanks, >> -John >> >> On Jun 12, 2013, at 6:11 PM, Mike Tutkowski >> wrote: >> >> I see, John. >> >> That is an interesting idea. >> >> We'd also have to change the storage allocator(s) to favor QoS-supported >> storage if those fields are filled in. >> >> >> On Wed, Jun 12, 2013 at 4:09 PM, John Burwell wrote= : >> >>> Mike, >>> >>> My thought is that we present the min/max IOPS fields for read/write >>> operations for all compute and disk offerings. When the VM is allocate= d, >>> we determine the best way to fulfill that QoS. It sounds like storage >>> level guarantees would always be preferred. If no storage is available= to >>> guarantee the QoS, the allocator falls back to using hypervisor. This >>> approach only works if summing the discrete read/write min/max values t= o >>> get to min/max total IOPS would be acceptable. >>> >>> Thanks, >>> -John >>> >>> >>> On Jun 12, 2013, at 6:03 PM, Mike Tutkowski < >>> mike.tutkowski@solidfire.com> wrote: >>> >>> I hate to say it, but I believe Storage QoS with a Min and Max will >>> always be optimal over hypervisor rate limiting. >>> >>> The only time you'd want to use hypervisor rate limiting is if storage >>> QoS is not available. >>> >>> We currently have no way to know what storage the root or data disk wil= l >>> be deployed to, so we have to present the fields in all situation. >>> >>> This is because of - in my opinion - the somewhat flawed way CloudStack >>> handles storage tags. You are free to enter in any text there and we do= n't >>> know what it maps to when the Compute or Disk Offering dialog is up. >>> >>> >>> On Wed, Jun 12, 2013 at 3:58 PM, John Burwell wrote= : >>> >>>> Mike, >>>> >>>> Please see my comments/questions in-line below. >>>> >>>> Thanks, >>>> -John >>>> >>>> On Jun 12, 2013, at 5:37 PM, Mike Tutkowski < >>>> mike.tutkowski@solidfire.com> wrote: >>>> >>>> Hi Wei, >>>> >>>> So, not entirely sure I follow. >>>> >>>> Will what I wrote earlier work? Here is a copy of what I wrote: >>>> >>>> Let's just called these radio buttons 1) Hypervisor QoS and 2) Storage >>>> QoS for the purpose of this e-mail. >>>> >>>> By default, neither radio button is selected and no QoS fields are >>>> visible. >>>> >>>> >>>> I prefer a None option selected by default. I find the no button >>>> selected in a Radio button group to be confusing. >>>> >>>> Also, do we have a mechanism to determine whether or not this radio >>>> button group should be displayed at all? If there are no devices or >>>> hypervisors configured to fulfill the QoS, we shouldn't mislead the us= er ... >>>> >>>> >>>> If the user selects the Storage QoS radio button, then the Custom IOPS >>>> checkbox and the Min and Max text fields are made visible. >>>> >>>> If the user changes his mind and selects the Hypervisor QoS radio >>>> button, then the Custom IOPS checkbox and the Min and Max text fields >>>> disappear and are replaced by the two Hypervisor QoS text fields. >>>> >>>> This way, the user can choose neither QoS option or one of them or the >>>> other, but never both. >>>> >>>> On the API side, I was thinking of having logic in place when a reques= t >>>> comes in to create a Disk Offering to confirm these fields are the way= we >>>> want them. >>>> >>>> Once we know the Disk Offering is in order, a user can create a data >>>> disk from it. Since we checked the validity of the Disk Offering when = it >>>> was created, the VM should never be asked to use Hypervisor QoS when >>>> Storage QoS in being utilized. >>>> >>>> >>>> Overall, this model sounds very reasonable. Wondering aloud: Would be >>>> possible to have the UI simply ask for a storage QoS, and let the syst= em >>>> pick the optional implementation. For example, if a VM is attached to= KVM >>>> and SolidFire is available, we chose to fulfill the QoS using the stor= age >>>> device. This approach would also allow us to fallback in the event th= at we >>>> can't allocate storage for the QoS, but we have hypervisor capacity. C= ould >>>> we always deterministically determine the optimal implementation of th= e QoS >>>> at time of VM deployment? >>>> >>>> >>>> Thanks! >>>> >>>> >>>> On Wed, Jun 12, 2013 at 3:33 PM, Wei ZHOU wrote= : >>>> >>>>> Mike, John >>>>> >>>>> The Disk I/O Throttling works like: >>>>> >>>>> (1) For User VM, >>>>> (1.1) root disks: service offering -> default value in global >>>>> configuration >>>>> (1.2) data disks: disk offering -> default value in global >>>>> configuration >>>>> >>>>> (2) For System VM >>>>> root disks: service offering >>>>> >>>>> -Wei >>>>> >>>>> >>>>> 2013/6/12 John Burwell >>>>> >>>>> > Mike, >>>>> > >>>>> > That is one possibility. The other possibility is that hypervisor = is >>>>> > going to throttle I/O on all disks attached. Therefore, we need >>>>> answers to >>>>> > the following questions: >>>>> > >>>>> > 1. Is I/O throttling applied to the root disk or all disks >>>>> > attached to the VM? >>>>> > 2. If I/O throttling is applied to all disks, how is the >>>>> > throttling distributed amongst the disks if only one read/write >>>>> value is >>>>> > defined? >>>>> > >>>>> > Thanks, >>>>> > -John >>>>> > >>>>> > On Jun 12, 2013, at 5:13 PM, Mike Tutkowski < >>>>> mike.tutkowski@solidfire.com> >>>>> > wrote: >>>>> > >>>>> > > Hey John, >>>>> > > >>>>> > > Perhaps I don't fully understand how Wei's feature works. >>>>> > > >>>>> > > I guess I thought if you choose Hypervisor QoS, you do so on >>>>> Compute >>>>> > > Offerings (root disks) and/or Disk Offerings (data disks). >>>>> > > >>>>> > > From my thinking, you're root disk could be under Hypervisor QoS, >>>>> but >>>>> > your >>>>> > > data disk could be under Storage QoS. >>>>> > > >>>>> > > Is that incorrect? >>>>> > > >>>>> > > Thanks >>>>> > > >>>>> > > >>>>> > > On Wed, Jun 12, 2013 at 3:10 PM, John Burwell >>>>> > wrote: >>>>> > > >>>>> > >> Mike, >>>>> > >> >>>>> > >> As I understand these two patches, the throttled I?O settings ar= e >>>>> > applied >>>>> > >> from the hypervisor side, and possibly defined in a compute >>>>> offering >>>>> > where >>>>> > >> provisioned IOPS are defined on the storage side through disk >>>>> > offerings. I >>>>> > >> don't see how the management server could enforce this mutual >>>>> exclusion >>>>> > >> between provisioned IOPS and throttled I/O until a compute and >>>>> disk >>>>> > >> offering were selected. These selections come together when we >>>>> deploy a >>>>> > >> VM. Based on these assumptions, I would expect to see >>>>> enforcement of >>>>> > this >>>>> > >> rule in UI at the time of VM creation/definition and on the serv= er >>>>> > side, as >>>>> > >> part of the VM creation. It feels like any attempt to enforce >>>>> this rule >>>>> > >> when defining offering would be premature, and unnecessarily lim= it >>>>> > >> flexibility. Are my assumptions and understanding correct? >>>>> > >> >>>>> > >> Thanks, >>>>> > >> -John >>>>> > >> >>>>> > >> On Jun 12, 2013, at 5:04 PM, Mike Tutkowski < >>>>> > mike.tutkowski@solidfire.com> >>>>> > >> wrote: >>>>> > >> >>>>> > >>> Hi John, >>>>> > >>> >>>>> > >>> So, maybe I'm wrong about this, but what I was thinking is that >>>>> we'd >>>>> > >> build >>>>> > >>> two radio buttons into the Add Disk Offering dialog (we can >>>>> ignore >>>>> > >> Compute >>>>> > >>> Offerings for 4.2 since my feature doesn't yet support them). >>>>> > >>> >>>>> > >>> Let's just called these radio buttons 1) Hypervisor QoS and 2) >>>>> Storage >>>>> > >> QoS >>>>> > >>> for the purpose of this e-mail. >>>>> > >>> >>>>> > >>> By default, neither radio button is selected and no QoS fields >>>>> are >>>>> > >> visible. >>>>> > >>> >>>>> > >>> If the user selects the Storage QoS radio button, then the >>>>> Custom IOPS >>>>> > >>> checkbox and the Min and Max text fields are made visible. >>>>> > >>> >>>>> > >>> If the user changes his mind and selects the Hypervisor QoS rad= io >>>>> > button, >>>>> > >>> then the Custom IOPS checkbox and the Min and Max text fields >>>>> disappear >>>>> > >> and >>>>> > >>> are replaced by the two Hypervisor QoS text fields. >>>>> > >>> >>>>> > >>> This way, the user can choose neither QoS option or one of them >>>>> or the >>>>> > >>> other, but never both. >>>>> > >>> >>>>> > >>> On the API side, I was thinking of having logic in place when a >>>>> request >>>>> > >>> comes in to create a Disk Offering to confirm these fields are >>>>> the way >>>>> > we >>>>> > >>> want them. >>>>> > >>> >>>>> > >>> Once we know the Disk Offering is in order, a user can create a >>>>> data >>>>> > disk >>>>> > >>> from it. Since we checked the validity of the Disk Offering whe= n >>>>> it was >>>>> > >>> created, the VM should never be asked to use Hypervisor QoS whe= n >>>>> > Storage >>>>> > >>> QoS in being utilized. >>>>> > >>> >>>>> > >>> Does that make sense or did I miss something? >>>>> > >>> >>>>> > >>> Thanks >>>>> > >>> >>>>> > >>> >>>>> > >>> On Wed, Jun 12, 2013 at 2:54 PM, John Burwell < >>>>> jburwell@basho.com> >>>>> > >> wrote: >>>>> > >>> >>>>> > >>>> Mike, >>>>> > >>>> >>>>> > >>>> Looking through the code, I am trying to understand how >>>>> > >>>> CreateDiskOfferingCmd would have the context to identify the >>>>> conflict. >>>>> > >>>> Naively, it seems to me that this rule would need to be >>>>> enforced when >>>>> > a >>>>> > >>>> virtual machine is being deployed. Looking through the code, >>>>> it seems >>>>> > >> like >>>>> > >>>> we should add a private validateStorageQoS method to >>>>> > >>>> com.cloud.vm.UserVmManagerImpl to check this condition and >>>>> throws a >>>>> > >>>> ResourceAllocationException when the QoS definitions are >>>>> inconsistent. >>>>> > >> We >>>>> > >>>> would then add calls to it from each of the VM creation method= s >>>>> in the >>>>> > >>>> service. Do this type of approach sound reasonable? >>>>> > >>>> >>>>> > >>>> Thanks, >>>>> > >>>> -John >>>>> > >>>> >>>>> > >>>> On Jun 12, 2013, at 4:30 PM, Mike Tutkowski < >>>>> > >> mike.tutkowski@solidfire.com> >>>>> > >>>> wrote: >>>>> > >>>> >>>>> > >>>>> Hi John, >>>>> > >>>>> >>>>> > >>>>> So, here's what I was planning to do. Of course feel free to >>>>> correct >>>>> > me >>>>> > >>>> on >>>>> > >>>>> this approach. >>>>> > >>>>> >>>>> > >>>>> I think it's OK if Wei merges his code into master and then I >>>>> can >>>>> > draw >>>>> > >>>> from >>>>> > >>>>> the main repo and merge master into mine locally. >>>>> > >>>>> >>>>> > >>>>> 1) Once I get Wei's code and merge, I plan to add a little GU= I >>>>> code >>>>> > to >>>>> > >>>> make >>>>> > >>>>> it user friendly (toggle between these features on the Add Di= sk >>>>> > >> Offering >>>>> > >>>>> window). >>>>> > >>>>> >>>>> > >>>>> 2) I plan to write validation logic for the >>>>> create-disk-offering API >>>>> > >>>>> command which throws an exception if the rules are not >>>>> followed (this >>>>> > >>>>> should never be triggered from the GUI since the GUI will hav= e >>>>> > controls >>>>> > >>>> in >>>>> > >>>>> place to toggle between the one feature and the other). >>>>> > >>>>> >>>>> > >>>>> I'm not sure about documentation. I haven't had much >>>>> experience with >>>>> > it >>>>> > >>>> on >>>>> > >>>>> CloudStack projects yet. >>>>> > >>>>> >>>>> > >>>>> Thanks! >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> On Wed, Jun 12, 2013 at 2:21 PM, John Burwell < >>>>> jburwell@basho.com> >>>>> > >>>> wrote: >>>>> > >>>>> >>>>> > >>>>>> Mike, >>>>> > >>>>>> >>>>> > >>>>>> Yes, these server-side rails need to be defined and >>>>> implemented >>>>> > before >>>>> > >>>>>> either patch can be merged. From my perspective, I would >>>>> like to >>>>> > see >>>>> > >>>> the >>>>> > >>>>>> rule implemented in the hypervisor as part of the validation >>>>> of the >>>>> > >>>> virtual >>>>> > >>>>>> machine definition. We also need to make sure that this >>>>> mutual >>>>> > >>>> exclusion >>>>> > >>>>>> is documented. Do we usually include this type of >>>>> documentation >>>>> > with >>>>> > >>>>>> patches of this nature? >>>>> > >>>>>> >>>>> > >>>>>> Thanks, >>>>> > >>>>>> -John >>>>> > >>>>>> >>>>> > >>>>>> On Jun 12, 2013, at 2:18 PM, Mike Tutkowski < >>>>> > >>>> mike.tutkowski@solidfire.com> >>>>> > >>>>>> wrote: >>>>> > >>>>>> >>>>> > >>>>>>> Currently they are not yet implemented. >>>>> > >>>>>>> >>>>> > >>>>>>> We have to make sure they are implemented in the GUI from a >>>>> > usability >>>>> > >>>>>>> standpoint, but the API must check for consistency and thro= w >>>>> an >>>>> > >>>> exception >>>>> > >>>>>>> if necessary. >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> On Wed, Jun 12, 2013 at 11:03 AM, John Burwell < >>>>> jburwell@basho.com >>>>> > > >>>>> > >>>>>> wrote: >>>>> > >>>>>>> >>>>> > >>>>>>>> Mike, >>>>> > >>>>>>>> >>>>> > >>>>>>>> Are the checks only implemented in the UI? >>>>> > >>>>>>>> >>>>> > >>>>>>>> Thanks, >>>>> > >>>>>>>> -John >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> On Jun 12, 2013, at 1:02 PM, Mike Tutkowski >>>>> > >>>>>>>> wrote: >>>>> > >>>>>>>> >>>>> > >>>>>>>>> Hi John, >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Wei and I have discussed making the two features mutually >>>>> > >> exclusive. >>>>> > >>>> We >>>>> > >>>>>>>>> agree with you that only one should be active at a time. >>>>> We plan >>>>> > to >>>>> > >>>>>>>>> implement in the GUI a mechanism (maybe radio buttons) to >>>>> turn >>>>> > his >>>>> > >>>>>>>> feature >>>>> > >>>>>>>>> on and mine off and vice versa. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> I was thinking if I wait until he checks in his code, the= n >>>>> I >>>>> > update >>>>> > >>>> and >>>>> > >>>>>>>>> merge that I will be the person resolving merge conflicts >>>>> in the >>>>> > >>>>>>>> JavaScript >>>>> > >>>>>>>>> code (there shouldn't be a problem in the Java code) as >>>>> opposed >>>>> > to >>>>> > >>>>>>>> putting >>>>> > >>>>>>>>> that work on someone else. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Let me know what you think. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Oh, was going to ask you what "FS" stands for here. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> Thanks! >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> On Wed, Jun 12, 2013 at 10:56 AM, John Burwell < >>>>> > jburwell@basho.com >>>>> > >>> >>>>> > >>>>>>>> wrote: >>>>> > >>>>>>>>> >>>>> > >>>>>>>>>> Mike, >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> How have Wei and you resolved the issue of conflicting Q= oS >>>>> > >>>> mechanisms >>>>> > >>>>>>>>>> between the Hypervisor and Storage layers? Have the >>>>> affected >>>>> > FSs >>>>> > >>>> been >>>>> > >>>>>>>>>> updated with that decision? >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> In terms of merge timing, can you describe the >>>>> dependencies >>>>> > >> between >>>>> > >>>>>> the >>>>> > >>>>>>>>>> patches? >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> Thanks, >>>>> > >>>>>>>>>> -John >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> On Jun 12, 2013, at 12:43 PM, Mike Tutkowski >>>>> > >>>>>>>>>> wrote: >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>>> No problem, John. >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> I still want to re-review it by myself before coming up >>>>> with a >>>>> > >> new >>>>> > >>>>>>>> patch >>>>> > >>>>>>>>>>> file. >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> Also, maybe I should first wait for Wei's changes to be >>>>> checked >>>>> > >> in >>>>> > >>>>>> and >>>>> > >>>>>>>>>>> merge those into mine before generating a new patch fil= e? >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> On Wed, Jun 12, 2013 at 10:40 AM, John Burwell < >>>>> > >> jburwell@basho.com >>>>> > >>>>> >>>>> > >>>>>>>>>> wrote: >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>>> Mike, >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> I just realized that I forgot to publish my review. I >>>>> am >>>>> > >> offline >>>>> > >>>>>> ATM, >>>>> > >>>>>>>>>>>> but I will publish it in the next couple of hours. >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> Do you plan to update your the patch in Review Board? >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> Sorry for the oversight, >>>>> > >>>>>>>>>>>> -John >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> On Jun 12, 2013, at 2:26 AM, Mike Tutkowski >>>>> > >>>>>>>>>>>> wrote: >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> Hi Edison, John, and Wei (and whoever else is reading >>>>> this :) >>>>> > >> ), >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> Just an FYI that I believe I have implemented all the >>>>> areas >>>>> > we >>>>> > >>>>>> wanted >>>>> > >>>>>>>>>>>>> addressed. >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> I plan to review the code again tomorrow morning or >>>>> > afternoon, >>>>> > >>>> then >>>>> > >>>>>>>>>> send >>>>> > >>>>>>>>>>>> in >>>>> > >>>>>>>>>>>>> another patch. >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> Thanks for all the work on this everyone! >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:29 PM, Mike Tutkowski < >>>>> > >>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> Sure, that sounds good. >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:11 PM, Wei ZHOU < >>>>> > >>>> ustcweizhou@gmail.com >>>>> > >>>>>>> >>>>> > >>>>>>>>>>>> wrote: >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> Hi Mike, >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> It looks the two feature do not have many conflicts >>>>> in Java >>>>> > >>>> code, >>>>> > >>>>>>>>>>>> except >>>>> > >>>>>>>>>>>>>>> the cloudstack UI. >>>>> > >>>>>>>>>>>>>>> If you do not mind, I will merge disk_io_throttling >>>>> branch >>>>> > >> into >>>>> > >>>>>>>>>> master >>>>> > >>>>>>>>>>>>>>> this >>>>> > >>>>>>>>>>>>>>> week, so that you can develop based on it. >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> -Wei >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> 2013/6/11 Mike Tutkowski < >>>>> mike.tutkowski@solidfire.com> >>>>> > >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> Hey John, >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> The SolidFire patch does not depend on the >>>>> object_store >>>>> > >>>> branch, >>>>> > >>>>>>>> but >>>>> > >>>>>>>>>> - >>>>> > >>>>>>>>>>>> as >>>>> > >>>>>>>>>>>>>>>> Edison mentioned - it might be easier if we merge >>>>> the >>>>> > >>>> SolidFire >>>>> > >>>>>>>>>> branch >>>>> > >>>>>>>>>>>>>>> into >>>>> > >>>>>>>>>>>>>>>> the object_store branch before object_store goes >>>>> into >>>>> > >> master. >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> I'm not sure how the disk_io_throttling fits into >>>>> this >>>>> > merge >>>>> > >>>>>>>>>> strategy. >>>>> > >>>>>>>>>>>>>>>> Perhaps Wei can chime in on that. >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 11:07 AM, John Burwell < >>>>> > >>>>>>>> jburwell@basho.com> >>>>> > >>>>>>>>>>>>>>> wrote: >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> Mike, >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> We have a delicate merge dance to perform. The >>>>> > >>>>>>>> disk_io_throttling, >>>>> > >>>>>>>>>>>>>>>>> solidfire, and object_store appear to have a >>>>> number of >>>>> > >>>>>>>> overlapping >>>>> > >>>>>>>>>>>>>>>>> elements. I understand the dependencies between >>>>> the >>>>> > >> patches >>>>> > >>>> to >>>>> > >>>>>>>> be >>>>> > >>>>>>>>>> as >>>>> > >>>>>>>>>>>>>>>>> follows: >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> object_store <- solidfire -> disk_io_throttling >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> Am I correct that the device management aspects o= f >>>>> > >> SolidFire >>>>> > >>>>>> are >>>>> > >>>>>>>>>>>>>>> additive >>>>> > >>>>>>>>>>>>>>>>> to the object_store branch or there are circular >>>>> > dependency >>>>> > >>>>>>>> between >>>>> > >>>>>>>>>>>>>>> the >>>>> > >>>>>>>>>>>>>>>>> branches? Once we understand the dependency >>>>> graph, we >>>>> > can >>>>> > >>>>>>>>>> determine >>>>> > >>>>>>>>>>>>>>> the >>>>> > >>>>>>>>>>>>>>>>> best approach to land the changes in master. >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> Thanks, >>>>> > >>>>>>>>>>>>>>>>> -John >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>> On Jun 10, 2013, at 11:10 PM, Mike Tutkowski < >>>>> > >>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> >>>>> > >>>>>>>>>>>>>>>>> wrote: >>>>> > >>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> Also, if we are good with Edison merging my code >>>>> into >>>>> > his >>>>> > >>>>>> branch >>>>> > >>>>>>>>>>>>>>> before >>>>> > >>>>>>>>>>>>>>>>>> going into master, I am good with that. >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> We can remove the StoragePoolType.Dynamic code >>>>> after his >>>>> > >>>> merge >>>>> > >>>>>>>> and >>>>> > >>>>>>>>>>>>>>> we >>>>> > >>>>>>>>>>>>>>>> can >>>>> > >>>>>>>>>>>>>>>>>> deal with Burst IOPS then, as well. >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 9:08 PM, Mike Tutkowski = < >>>>> > >>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> Let me make sure I follow where we're going her= e: >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 1) There should be NO references to hypervisor >>>>> code in >>>>> > >> the >>>>> > >>>>>>>>>> storage >>>>> > >>>>>>>>>>>>>>>>>>> plug-ins code (this includes the default storag= e >>>>> > plug-in, >>>>> > >>>>>> which >>>>> > >>>>>>>>>>>>>>>>> currently >>>>> > >>>>>>>>>>>>>>>>>>> sends several commands to the hypervisor in use >>>>> > (although >>>>> > >>>> it >>>>> > >>>>>>>> does >>>>> > >>>>>>>>>>>>>>> not >>>>> > >>>>>>>>>>>>>>>>> know >>>>> > >>>>>>>>>>>>>>>>>>> which hypervisor (XenServer, ESX(i), etc.) is >>>>> actually >>>>> > in >>>>> > >>>>>> use)) >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 2) managed=3Dtrue or managed=3Dfalse can be pla= ced >>>>> in the >>>>> > url >>>>> > >>>>>> field >>>>> > >>>>>>>>>> (if >>>>> > >>>>>>>>>>>>>>>> not >>>>> > >>>>>>>>>>>>>>>>>>> present, we default to false). This info is >>>>> stored in >>>>> > the >>>>> > >>>>>>>>>>>>>>>>>>> storage_pool_details table. >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 3) When the "attach" command is sent to the >>>>> hypervisor >>>>> > in >>>>> > >>>>>>>>>>>>>>> question, we >>>>> > >>>>>>>>>>>>>>>>>>> pass the managed property along (this takes the >>>>> place >>>>> > of >>>>> > >>>> the >>>>> > >>>>>>>>>>>>>>>>>>> StoragePoolType.Dynamic check). >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 4) execute(AttachVolumeCommand) in the hypervis= or >>>>> > checks >>>>> > >>>> for >>>>> > >>>>>>>> the >>>>> > >>>>>>>>>>>>>>>> managed >>>>> > >>>>>>>>>>>>>>>>>>> property. If true for an attach, the necessary >>>>> > hypervisor >>>>> > >>>>>> data >>>>> > >>>>>>>>>>>>>>>>> structure is >>>>> > >>>>>>>>>>>>>>>>>>> created and the rest of the attach command >>>>> executes to >>>>> > >>>> attach >>>>> > >>>>>>>> the >>>>> > >>>>>>>>>>>>>>>>> volume. >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 5) When execute(AttachVolumeCommand) is invoked >>>>> to >>>>> > >> detach a >>>>> > >>>>>>>>>> volume, >>>>> > >>>>>>>>>>>>>>>> the >>>>> > >>>>>>>>>>>>>>>>>>> same check is made. If managed, the hypervisor >>>>> data >>>>> > >>>> structure >>>>> > >>>>>>>> is >>>>> > >>>>>>>>>>>>>>>>> removed. >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> 6) I do not see an clear way to support Burst >>>>> IOPS in >>>>> > 4.2 >>>>> > >>>>>>>> unless >>>>> > >>>>>>>>>>>>>>> it is >>>>> > >>>>>>>>>>>>>>>>>>> stored in the volumes and disk_offerings table. >>>>> If we >>>>> > >> have >>>>> > >>>>>> some >>>>> > >>>>>>>>>>>>>>> idea, >>>>> > >>>>>>>>>>>>>>>>>>> that'd be cool. >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> Thanks! >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 8:58 PM, Mike Tutkowski= < >>>>> > >>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote: >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>>> "+1 -- Burst IOPS can be implemented while >>>>> avoiding >>>>> > >>>>>>>>>> implementation >>>>> > >>>>>>>>>>>>>>>>>>>> attributes. I always wondered about the detai= ls >>>>> > field. >>>>> > >> I >>>>> > >>>>>>>> think >>>>> > >>>>>>>>>>>>>>> we >>>>> > >>>>>>>>>>>>>>>>> should >>>>> > >>>>>>>>>>>>>>>>>>>> beef up the description in the documentation >>>>> regarding >>>>> > >> the >>>>> > >>>>>>>>>>>>>>> expected >>>>> > >>>>>>>>>>>>>>>>> format >>>>> > >>>>>>>>>>>>>>>>>>>> of the field. In 4.1, I noticed that the >>>>> details are >>>>> > >> not >>>>> > >>>>>>>>>>>>>>> returned on >>>>> > >>>>>>>>>>>>>>>>> the >>>>> > >>>>>>>>>>>>>>>>>>>> createStoratePool updateStoragePool, or >>>>> > listStoragePool >>>>> > >>>>>>>>>> response. >>>>> > >>>>>>>>>>>>>>>> Why >>>>> > >>>>>>>>>>>>>>>>>>>> don't we return it? It seems like it would be >>>>> useful >>>>> > >> for >>>>> > >>>>>>>>>> clients >>>>> > >>>>>>>>>>>>>>> to >>>>> > >>>>>>>>>>>>>>>> be >>>>> > >>>>>>>>>>>>>>>>>>>> able to inspect the contents of the details >>>>> field." >>>>> > >>>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>>> Not sure how this would work storing Burst IOP= S >>>>> here. >>>>> > >>>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>>> Burst IOPS need to be variable on a Disk >>>>> > >> Offering-by-Disk >>>>> > >>>>>>>>>> Offering >>>>> > >>>>>>>>>>>>>>>>>>>> basis. For each Disk Offering created, you hav= e >>>>> to be >>>>> > >> able >>>>> > >>>>>> to >>>>> > >>>>>>>>>>>>>>>> associate >>>>> > >>>>>>>>>>>>>>>>>>>> unique Burst IOPS. There is a >>>>> disk_offering_details >>>>> > >> table. >>>>> > >>>>>>>> Maybe >>>>> > >>>>>>>>>>>>>>> it >>>>> > >>>>>>>>>>>>>>>>> could >>>>> > >>>>>>>>>>>>>>>>>>>> go there? >>>>> > >>>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>>> I'm also not sure how you would accept the >>>>> Burst IOPS >>>>> > in >>>>> > >>>> the >>>>> > >>>>>>>> GUI >>>>> > >>>>>>>>>>>>>>> if >>>>> > >>>>>>>>>>>>>>>>> it's >>>>> > >>>>>>>>>>>>>>>>>>>> not stored like the Min and Max fields are in >>>>> the DB. >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>>> -- >>>>> > >>>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud< >>>>> > >>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dp= lay >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>> *=99* >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>>>> -- >>>>> > >>>>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>>>>>>>>> Advancing the way the world uses the >>>>> > >>>>>>>>>>>>>>>>>> cloud< >>>>> > http://solidfire.com/solution/overview/?video=3Dplay> >>>>> > >>>>>>>>>>>>>>>>>> *=99* >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>>> -- >>>>> > >>>>>>>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>>>>>>> Advancing the way the world uses the >>>>> > >>>>>>>>>>>>>>>> cloud< >>>>> http://solidfire.com/solution/overview/?video=3Dplay> >>>>> > >>>>>>>>>>>>>>>> *=99* >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> -- >>>>> > >>>>>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>>>>> Advancing the way the world uses the cloud< >>>>> > >>>>>>>>>>>> http://solidfire.com/solution/overview/?video=3Dplay> >>>>> > >>>>>>>>>>>>>> *=99* >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>> -- >>>>> > >>>>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>>>> Advancing the way the world uses the >>>>> > >>>>>>>>>>>>> cloud< >>>>> http://solidfire.com/solution/overview/?video=3Dplay> >>>>> > >>>>>>>>>>>>> *=99* >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> -- >>>>> > >>>>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>>>> Advancing the way the world uses the >>>>> > >>>>>>>>>>> cloud>>>> > >>>>> > >>>>>>>>>>> *=99* >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> -- >>>>> > >>>>>>>>> *Mike Tutkowski* >>>>> > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>>>> o: 303.746.7302 >>>>> > >>>>>>>>> Advancing the way the world uses the >>>>> > >>>>>>>>> cloud >>>>> > >>>>>>>>> *=99* >>>>> > >>>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> -- >>>>> > >>>>>>> *Mike Tutkowski* >>>>> > >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>>>> o: 303.746.7302 >>>>> > >>>>>>> Advancing the way the world uses the >>>>> > >>>>>>> cloud >>>>> > >>>>>>> *=99* >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> -- >>>>> > >>>>> *Mike Tutkowski* >>>>> > >>>>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>>>> e: mike.tutkowski@solidfire.com >>>>> > >>>>> o: 303.746.7302 >>>>> > >>>>> Advancing the way the world uses the >>>>> > >>>>> cloud >>>>> > >>>>> *=99* >>>>> > >>>> >>>>> > >>>> >>>>> > >>> >>>>> > >>> >>>>> > >>> -- >>>>> > >>> *Mike Tutkowski* >>>>> > >>> *Senior CloudStack Developer, SolidFire Inc.* >>>>> > >>> e: mike.tutkowski@solidfire.com >>>>> > >>> o: 303.746.7302 >>>>> > >>> Advancing the way the world uses the >>>>> > >>> cloud >>>>> > >>> *=99* >>>>> > >> >>>>> > >> >>>>> > > >>>>> > > >>>>> > > -- >>>>> > > *Mike Tutkowski* >>>>> > > *Senior CloudStack Developer, SolidFire Inc.* >>>>> > > e: mike.tutkowski@solidfire.com >>>>> > > o: 303.746.7302 >>>>> > > Advancing the way the world uses the >>>>> > > cloud >>>>> > > *=99* >>>>> > >>>>> > >>>>> >>>> >>>> >>>> >>>> -- >>>> *Mike Tutkowski* >>>> *Senior CloudStack Developer, SolidFire Inc.* >>>> e: mike.tutkowski@solidfire.com >>>> o: 303.746.7302 >>>> Advancing the way the world uses the cloud >>>> *=99* >>>> >>>> >>>> >>> >>> >>> -- >>> *Mike Tutkowski* >>> *Senior CloudStack Developer, SolidFire Inc.* >>> e: mike.tutkowski@solidfire.com >>> o: 303.746.7302 >>> Advancing the way the world uses the cloud >>> *=99* >>> >>> >>> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkowski@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the cloud >> *=99* >> >> >> > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkowski@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the cloud > *=99* > --=20 *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkowski@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud *=99* --001a11c2deaa75abbd04defc9181--