cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: [MERGE] disk_io_throttling to MASTER
Date Fri, 14 Jun 2013 17:35:27 GMT
Hi John,

Are you thinking we add a column on to the storage pool table, IOPS_Count,
where we add and subtract committed IOPS?

That is easy enough.

How do you want to determine what the SAN is capable of supporting IOPS
wise? Remember we're dealing with a dynamic SAN here...as you add storage
nodes to the cluster, the number of IOPS increases. Do we have a thread we
can use to query external devices like this SAN to update the supported
number of IOPS?

Also, how do you want to enforce the IOPS limit? Do we pass in an
overcommit ration to the plug-in when it's created? We would need to store
this in the storage_pool table, as well, I believe.

We should also get Wei involved in this as his feature will need similar
functionality.

Also, we should do this FAST as we have only two weeks left and many of us
will be out for several days for the CS Collab Conference.

Thanks


On Fri, Jun 14, 2013 at 10:46 AM, John Burwell <jburwell@basho.com> wrote:

> Mike,
>
> Querying the SAN only indicates the number of IOPS currently in use.  The
> allocator needs to check the number of IOPS committed which is tracked by
> CloudStack.  For 4.2, we should be able to query the number of IOPS
> committed to a DataStore, and determine whether or not the number requested
> can be fulfilled by that device.  It seems to be that a
> DataStore#getCommittedIOPS() : Long method would be sufficient.
>  DataStore's that don't support provisioned IOPS would return null.
>
> As I mentioned previously, I am very reluctant for any feature to come
> into master that can exhaust resources.
>
> Thanks,
> -John
>
> On Jun 13, 2013, at 9:27 PM, Mike Tutkowski <mike.tutkowski@solidfire.com>
> wrote:
>
> > Yeah, I'm not sure I could come up with anything near an accurate
> > assessment of how many IOPS are currently available on the SAN (or even a
> > total number that are available for volumes). Not sure if there's yet an
> > API call for that.
> >
> > If I did know this number (total number of IOPS supported by the SAN),
> we'd
> > still have to keep track of the total number of volumes we've created
> from
> > CS on the SAN in terms of their IOPS. Also, if an admin issues an API
> call
> > directly to the SAN to tweak the number of IOPS on a given volume or set
> of
> > volumes (not supported from CS, but supported via the SolidFire API), our
> > numbers in CS would be off.
> >
> > I'm thinking verifying sufficient number of IOPS is a really good idea
> for
> > a future release.
> >
> > For 4.2 I think we should stick to having the allocator detect if storage
> > QoS is desired and if the storage pool in question supports that feature.
> >
> > If you really are over provisioned on your SAN in terms of IOPS or
> > capacity, the SAN can let the admin know in several different ways
> (e-mail,
> > SNMP, GUI).
> >
> >
> > On Thu, Jun 13, 2013 at 7:02 PM, John Burwell <jburwell@basho.com>
> wrote:
> >
> >> Mike,
> >>
> >> Please see my comments in-line below.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Jun 13, 2013, at 6:09 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> >> wrote:
> >>
> >>> Comments below in red.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Thu, Jun 13, 2013 at 3:58 PM, John Burwell <jburwell@basho.com>
> >> wrote:
> >>>
> >>>> Mike,
> >>>>
> >>>> Overall, I agree with the steps to below for 4.2.  However, we may
> want
> >> to
> >>>> throw an exception if we can not fulfill a requested QoS.  If the user
> >> is
> >>>> expecting that the hypervisor will provide a particular QoS, and that
> is
> >>>> not possible, it seems like we should inform them rather than silently
> >>>> ignoring their request.
> >>>>
> >>>
> >>> Sure, that sounds reasonable.
> >>>
> >>> We'd have to come up with some way for the allocators to know about the
> >>> requested storage QoS and then query the candidate drivers.
> >>>
> >>> Any thoughts on how we might do that?
> >>>
> >>>
> >>>>
> >>>> To collect my thoughts from previous parts of the thread, I am
> >>>> uncomfortable with the idea that the management server can overcommit
> a
> >>>> resource.  You had mentioned querying the device for available IOPS.
> >> While
> >>>> that would be nice, it seems like we could fall back to a max IOPS and
> >>>> overcommit factor manually calculated and entered by the
> >>>> administrator/operator.  I think such threshold and allocation rails
> >> should
> >>>> be added for both provisioned IOPS and throttled I/O -- it is a basic
> >>>> feature of any cloud orchestration platform.
> >>>>
> >>>
> >>> Are you thinking this ability would make it into 4.2? Just curious what
> >>> release we're talking about here. For the SolidFire SAN, you might
> have,
> >>> say, four separate storage nodes to start (200,000 IOPS) and then later
> >> add
> >>> a new node (now you're at 250,000 IOPS). CS would have to have a way to
> >>> know that the number of supported IOPS has increased.
> >>
> >> Yes, I think we need some *basic*/conservative rails in 4.2.  For
> example,
> >> we may only support expanding capacity in 4.2, and not handle any
> >> re-balance scenarios --  node failure, addition, etc.   Extrapolating a
> >> bit, the throttled I/O enhancement seems like it needs a similar set of
> >> rails defined per host.
> >>
> >>>
> >>>
> >>>>
> >>>> For 4.3, I don't like the idea that a QoS would be expressed in a
> >>>> implementation specific manner.  I think we need to arrive at a
> general
> >>>> model that can be exploited by the allocators and planners.  We should
> >>>> restrict implementation specific key-value pairs (call them details,
> >>>> extended data, whatever) to information that is unique to the driver
> and
> >>>> would provide no useful information to the management server's
> >>>> orchestration functions.  A resource QoS does not seem to fit those
> >>>> criteria.
> >>>>
> >>>
> >>> I wonder if this would be a good discussion topic for Sunday's CS
> Collab
> >>> Conf hack day that Joe just sent out an e-mail about?
> >>
> >> Yes, it would -- I will put something in the wiki topic.  It will also
> be
> >> part of my talk on Monday -- How to Run from Zombie which include some
> of
> >> my opinions on the topic.
> >>
> >>>
> >>>
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Jun 13, 2013, at 5:44 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com>
> >>>> wrote:
> >>>>
> >>>>> So, here's my suggestion for 4.2:
> >>>>>
> >>>>> Accept the values as they are currently required (four new fields for
> >>>> Wei's
> >>>>> feature or two new fields for mine).
> >>>>>
> >>>>> The Add Disk Offering dialog needs three new radio buttons:
> >>>>>
> >>>>> 1) No QoS
> >>>>>
> >>>>> 2) Hypervisor QoS
> >>>>>
> >>>>> 3) Storage Qos
> >>>>>
> >>>>> The admin needs to specify storage tags that only map to storage that
> >>>>> supports Storage QoS.
> >>>>>
> >>>>> The admin needs to be aware for Hypervisor QoS that unless all
> >>>> hypervisors
> >>>>> in use support the new fields, they may not be enforced.
> >>>>>
> >>>>> Post 4.3:
> >>>>>
> >>>>> Come up with a way to more generally enter these parameters (probably
> >>>> just
> >>>>> key/value pairs sent to the drivers).
> >>>>>
> >>>>> Have the drivers expose their feature set so the allocators can
> >> consider
> >>>>> them more fully and throw an exception if there is not a sufficient
> >>>> match.
> >>>>>
> >>>>>
> >>>>> On Thu, Jun 13, 2013 at 3:31 PM, Mike Tutkowski <
> >>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>
> >>>>>> My thinking is, for 4.2, while not ideal, we will need to put some
> >>>> burden
> >>>>>> on the admin to configure a Disk Offering in a way that makes sense.
> >> For
> >>>>>> example, if he wants to use storage QoS with supported Min and Max
> >>>> values,
> >>>>>> he'll have to put in a storage tag that references the SolidFire
> >> primary
> >>>>>> storage (plug-in). If he puts in a storage tag that doesn't, then
> he's
> >>>> not
> >>>>>> going to get the Min and Max feature. We could add help text to the
> >>>> pop-up
> >>>>>> dialog that's displayed when you click in the Min and Max text
> fields.
> >>>>>>
> >>>>>> Same idea for Wei's feature.
> >>>>>>
> >>>>>> Not idea, true...perhaps we can brainstorm on a more comprehensive
> >>>>>> approach for post 4.2.
> >>>>>>
> >>>>>> Maybe in the future we could have the drivers advertise their
> >>>> capabilities
> >>>>>> and if the allocator feels a request is not being satisfied (say Min
> >> was
> >>>>>> entered, but it not's supported by any storage plug-in) it can throw
> >> an
> >>>>>> exception.
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jun 13, 2013 at 3:19 PM, Mike Tutkowski <
> >>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>
> >>>>>>> Comments below in red.
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jun 13, 2013 at 2:54 PM, John Burwell <jburwell@basho.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Mike,
> >>>>>>>>
> >>>>>>>> Please see my comment in-line below.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> -John
> >>>>>>>>
> >>>>>>>> On Jun 13, 2013, at 1:22 AM, Mike Tutkowski <
> >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi John,
> >>>>>>>>>
> >>>>>>>>> I've put comments below in red.
> >>>>>>>>>
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Jun 12, 2013 at 10:51 PM, John Burwell <
> jburwell@basho.com
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Mike,
> >>>>>>>>>>
> >>>>>>>>>> First and foremost, we must ensure that these two features are
> >>>>>>>> mutually
> >>>>>>>>>> exclusive in 4.2.  We don't want to find a configuration that
> >>>>>>>> contains both
> >>>>>>>>>> hypervisor and storage IOPS guarantees that leads to
> >>>> non-deterministic
> >>>>>>>>>> operations.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Agreed
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Restricting QoS expression to be either hypervisor or storage
> >>>> oriented
> >>>>>>>>>> solves the problem in short term.  As I understand storage tags,
> >> we
> >>>>>>>> have no
> >>>>>>>>>> means of expressing this type of mutual exclusion.  I wasn't
> >>>>>>>> necessarily
> >>>>>>>>>> intending that we implement this allocation model in 4.3, but
> >>>> instead,
> >>>>>>>>>> determine if this type model would be one we would want to
> support
> >>>> in
> >>>>>>>> the
> >>>>>>>>>> future.  If so, I would encourage us to ensure that the data
> model
> >>>> and
> >>>>>>>>>> current implementation would not preclude evolution in that
> >>>>>>>> direction.  My
> >>>>>>>>>> view is that this type of allocation model is what user's expect
> >> of
> >>>>>>>> "cloud"
> >>>>>>>>>> systems -- selecting the best available resource set to fulfill
>  a
> >>>>>>>> set of
> >>>>>>>>>> system requirements.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I believe we have meet your requirement here in that what we've
> >>>>>>>> implemented
> >>>>>>>>> should not make refinement difficult in the future. If we don't
> >>>> modify
> >>>>>>>>> allocators for 4.2, but we do for 4.3, we've made relatively
> simple
> >>>>>>>> changes
> >>>>>>>>> to enhance the current functioning of the system.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Looking through both patches, I have to say that the aggregated
> >> result
> >>>>>>>> seems a bit confusing.  There are six new attributes for throttled
> >>>> I/O and
> >>>>>>>> two for provisioned IOPS with no obvious grouping.  My concern is
> >> not
> >>>>>>>> technical, but rather, about maintainability.   At minimum,
> Javadoc
> >>>> should
> >>>>>>>> be added explaining the two sets of attributes and their mutual
> >>>> exclusion.
> >>>>>>>>
> >>>>>>>
> >>>>>>> I agree: We need JavaDoc to explain them and their mutual
> exclusion.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> The other part that is interesting is that throttled I/O provides
> >> both
> >>>>>>>> an IOPS and byte measured QoS as a rate where provisioned IOPS
> uses
> >> a
> >>>>>>>> range.  In order to select the best available resource to fulfill
> a
> >>>> QoS, we
> >>>>>>>> would need to have the QoS expression normalized in terms of units
> >>>> (IOPS or
> >>>>>>>> bytes) and their expression (rate vs. range).  If we want to
> >> achieve a
> >>>>>>>> model like I described, I think we would need to resolve this
> issue
> >>>> in 4.2
> >>>>>>>> to ensure a viable migration path.
> >>>>>>>
> >>>>>>>
> >>>>>>> I think we're not likely to be able to normalize the input for 4.2.
> >>>> Plus
> >>>>>>> people probably want to input the data in terms they're familiar
> with
> >>>> for
> >>>>>>> the products in question.
> >>>>>>>
> >>>>>>> Ideally we would fix the way we do storage tagging and let the user
> >>>> send
> >>>>>>> key/value pairs to each vendor that could be selected due to a
> given
> >>>>>>> storage tag. I'm still not sure that would solve it because what
> >>>> happens if
> >>>>>>> you change the storage tag of a given Primary Storage after having
> >>>> created
> >>>>>>> a Disk Offering?
> >>>>>>>
> >>>>>>> Basically storage tagging is kind of a mess and we should re-think
> >> it.
> >>>>>>>
> >>>>>>> Also, we need to have a way for the drivers to expose their
> supported
> >>>>>>> feature sets so the allocators can make good choices.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> As I think through the implications of these requirements and
> >>>> reflect
> >>>>>>>> on
> >>>>>>>>>> the reviews, I don't understand why they haven't already
> impacted
> >>>> the
> >>>>>>>>>> allocators and planners.  As it stands, the current provisioned
> >> IOPS
> >>>>>>>> has no
> >>>>>>>>>> checks to ensure that the volumes are allocated to devices that
> >> have
> >>>>>>>>>> capacity to fulfill the requested QoS.  Therefore, as I
> understand
> >>>> the
> >>>>>>>>>> current patch, we can overcommit storage resources --
> potentially
> >>>>>>>> causing
> >>>>>>>>>> none of the QoS obligations from being fulfilled.  It seems to
> me
> >>>>>>>> that a
> >>>>>>>>>> DataStore supporting provisioned IOPS should express the maximum
> >>>> IOPS
> >>>>>>>> which
> >>>>>>>>>> it can support and some type of overcommitment factor.  This
> >>>>>>>> information
> >>>>>>>>>> should be used by the storage allocators to determine the device
> >>>> best
> >>>>>>>> able
> >>>>>>>>>> to support the resources needs of a volume.  It seems that a
> >> similar
> >>>>>>>> set of
> >>>>>>>>>> considerations would need to be added to the Hypervisor layer
> >> which
> >>>>>>>>>> allocating a VM to a host to prevent oversubscription.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Yeah, for this first release, we just followed the path that was
> >>>>>>>> previously
> >>>>>>>>> established for other properties you see on dialogs in CS: Just
> >>>> because
> >>>>>>>>> they're there doesn't mean the vendor your VM is deployed to
> >> supports
> >>>>>>>> them.
> >>>>>>>>> It is then up to the admin to make sure he inputs, say, a storage
> >> tag
> >>>>>>>> that
> >>>>>>>>> confines the deployment only to storage that supports the
> selected
> >>>>>>>>> features. This is not ideal, but it's kind of the way CloudStack
> >>>> works
> >>>>>>>>> today.
> >>>>>>>>
> >>>>>>>> I understand the tag functionality, and the need for the
> >> administrator
> >>>>>>>> to very carefully construct offerings.  My concern is that we over
> >>>>>>>> guarantee a resource's available IOPS.  For the purposes of
> >>>> illustration,
> >>>>>>>> let's say we have a SolidFire, and the max IOPS for that device is
> >>>> 100000.
> >>>>>>>> We also know that we can safely oversubscribe by 50%.  Therefore,
> >> we
> >>>>>>>> need to ensure that we don't allocate more than 150,000 guaranteed
> >>>> IOPS
> >>>>>>>> from that device.  Intuitively, it seems like the DataStore
> >>>> configuration
> >>>>>>>> should have a max assignable IOPS and overcommitment factor.  As
> we
> >>>>>>>> allocate volumes and attach VMs, we need to ensure that guarantee
> >>>> more IOPS
> >>>>>>>> exceed the configured maximum for a DataStore.  Does that make
> >> sense?
> >>>>>>>>
> >>>>>>>
> >>>>>>> I think that's a good idea for a future enhancement. I'm not even
> >> sure
> >>>> I
> >>>>>>> can query the SAN to find out how many IOPS safely remain. I'd have
> >> to
> >>>> get
> >>>>>>> all of the min values for all of the volumes on the SAN and total
> >> them
> >>>> up,
> >>>>>>> I suppose, and subtract it from the total (user facing) supported
> >> IOPS
> >>>> of
> >>>>>>> the system.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Another question occurs to me -- should we allow non-QoS
> resources
> >>>> to
> >>>>>>>> be
> >>>>>>>>>> assigned to hosts/storage devices that ensure QoS?  For
> >> provisioned
> >>>>>>>> IOPS, I
> >>>>>>>>>> think a side effect of the current implementation is SolidFire
> >>>> volumes
> >>>>>>>>>> always have a QoS.  However, for hypervisor throttled I/O, it
> >> seems
> >>>>>>>>>> entirely possible for non-QoS VMs to allocated side-by-side with
> >> QoS
> >>>>>>>> VMs.
> >>>>>>>>>> In this scenario, a greedy, unbounded VM could potentially
> starve
> >>>> out
> >>>>>>>> the
> >>>>>>>>>> other VMs on the host -- preventing the QoSes defined the
> >> collocated
> >>>>>>>> VMs
> >>>>>>>>>> from being fulfilled.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> You can make SolidFire volumes (inside and outside of CS) and not
> >>>>>>>> specify
> >>>>>>>>> IOPS. You'll still get guaranteed IOPS, but it will be at the
> >>>> defaults
> >>>>>>>> we
> >>>>>>>>> choose. Unless you over-provision IOPS on a SolidFire SAN, you
> will
> >>>>>>>> have
> >>>>>>>>> your Mins met.
> >>>>>>>>>
> >>>>>>>>> It sounds like you're perhaps looking for a storage tags
> exclusions
> >>>>>>>> list,
> >>>>>>>>> which might be nice to have at some point (i.e. don't deploy my
> >>>> volume
> >>>>>>>> to
> >>>>>>>>> storage with these following tags).
> >>>>>>>>
> >>>>>>>> I don't like the idea of a storage tags exclusion list as it would
> >>>>>>>> complicate component assembly.  It would require a storage plugin
> to
> >>>>>>>> anticipate all of the possible component assemblies and determine
> >> the
> >>>>>>>> invalid relationships.  I prefer that drivers express their
> >>>> capabilities
> >>>>>>>> which can be matched to a set of requested requirements.
> >>>>>>>>
> >>>>>>>
> >>>>>>> I'm not sure why a storage plug-in would care about inclusion or
> >>>>>>> exclusion lists. It just needs to advertise its functionality in a
> >> way
> >>>> the
> >>>>>>> allocator understands.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I agree with your assessment of Hypervisor QoS. Since it only
> >> limits
> >>>>>>>> IOPS,
> >>>>>>>>> it does not solve the Noisy Neighbor problem. Only a system with
> >>>>>>>> guaranteed
> >>>>>>>>> minimum IOPS does this.
> >>>>>>>>
> >>>>>>>> As I said, for SolidFire, it sounds like this problem does not
> >> exist.
> >>>>>>>> However, I am concerned with the more general case as we supported
> >>>> more
> >>>>>>>> devices with provisioned IOPS.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Post 4.2 we need to investigate a way to pass vendor-specific
> values
> >> to
> >>>>>>> drivers. Min and Max and pretty industry standard for provisioned
> >>>> IOPS, but
> >>>>>>> what if you break them out by read and write or do something else?
> We
> >>>> need
> >>>>>>> a more general mechanism.
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> In my opinion,  we need to ensure that hypervisor throttled I/O
> >> and
> >>>>>>>>>> storage provisioned IOPS are mutually exclusive per volume.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Agreed
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> We also need to understand the implications of these QoS
> >> guarantees
> >>>> on
> >>>>>>>>>> operation of the system to ensure that the underlying hardware
> >>>>>>>> resources
> >>>>>>>>>> can fulfill them.  Given the time frame, we will likely be
> forced
> >> to
> >>>>>>>> make
> >>>>>>>>>> compromises to achieve these goals, and refine the
> implementation
> >> in
> >>>>>>>> 4.3.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I agree, John. I also think you've come up with some great ideas
> >> for
> >>>>>>>> 4.3. :)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> -John
> >>>>>>>>>>
> >>>>>>>>>> On Jun 12, 2013, at 11:35 PM, Mike Tutkowski <
> >>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> Yeah, Alex, I think that's the way we were planning (with
> storage
> >>>>>>>> tags).
> >>>>>>>>>> I
> >>>>>>>>>>> believe John was just throwing out an idea that - in addition
> to
> >>>>>>>> storage
> >>>>>>>>>>> tags - we could look into these allocators (storage QoS being
> >>>>>>>> preferred,
> >>>>>>>>>>> then hypervisor QoS if storage QoS is not available, but
> >> hypervisor
> >>>>>>>> QoS
> >>>>>>>>>> is).
> >>>>>>>>>>>
> >>>>>>>>>>> I think John's concern is that you can enter in values for
> Wei's
> >>>> and
> >>>>>>>> my
> >>>>>>>>>>> feature that are not honored by other vendors (at least yet),
> so
> >> he
> >>>>>>>> was
> >>>>>>>>>>> hoping - in addition to storage tags - for the allocators to
> >> prefer
> >>>>>>>> these
> >>>>>>>>>>> vendors when these fields are filled in. As it stands today in
> >>>>>>>>>> CloudStack,
> >>>>>>>>>>> we already have this kind of an issue with other features
> (fields
> >>>> in
> >>>>>>>>>>> dialogs for features that not all vendors support). Perhaps
> post
> >>>> 4.2
> >>>>>>>> we
> >>>>>>>>>>> could look into generic name/value pairs (this is how OpenStack
> >>>>>>>> addresses
> >>>>>>>>>>> the issue).
> >>>>>>>>>>>
> >>>>>>>>>>> Honestly, I think we're too late in the game (two weeks until
> >> code
> >>>>>>>>>> freeze)
> >>>>>>>>>>> to go too deeply down that path in 4.2.
> >>>>>>>>>>>
> >>>>>>>>>>> It's probably better if we - at least for 4.2 - keep Wei's
> fields
> >>>>>>>> and my
> >>>>>>>>>>> fields as is, make sure only one or the other feature has data
> >>>>>>>> entered
> >>>>>>>>>> for
> >>>>>>>>>>> it (or neither), and call it good for 4.2.
> >>>>>>>>>>>
> >>>>>>>>>>> Then let's step back and look into a more general-purpose
> design
> >>>>>>>> that can
> >>>>>>>>>>> be applied throughout CloudStack where we have these
> situations.
> >>>>>>>>>>>
> >>>>>>>>>>> What do you think?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jun 12, 2013 at 5:21 PM, John Burwell <
> >> jburwell@basho.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Mike,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I just published my review @
> >> https://reviews.apache.org/r/11479/.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I apologize for the delay,
> >>>>>>>>>>>> -John
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Jun 12, 2013, at 12:43 PM, Mike Tutkowski <
> >>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> No problem, John.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I still want to re-review it by myself before coming up with
> a
> >>>> new
> >>>>>>>>>> patch
> >>>>>>>>>>>>> file.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, maybe I should first wait for Wei's changes to be
> checked
> >>>> in
> >>>>>>>> and
> >>>>>>>>>>>>> merge those into mine before generating a new patch file?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jun 12, 2013 at 10:40 AM, John Burwell <
> >>>> jburwell@basho.com
> >>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I just realized that I forgot to publish my review.  I am
> >>>> offline
> >>>>>>>> ATM,
> >>>>>>>>>>>>>> but I will publish it in the next couple of hours.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you plan to update your the patch in Review Board?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry for the oversight,
> >>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Jun 12, 2013, at 2:26 AM, Mike Tutkowski
> >>>>>>>>>>>>>> <mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Edison, John, and Wei (and whoever else is reading this
> :)
> >>>> ),
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Just an FYI that I believe I have implemented all the areas
> >> we
> >>>>>>>> wanted
> >>>>>>>>>>>>>>> addressed.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I plan to review the code again tomorrow morning or
> >> afternoon,
> >>>>>>>> then
> >>>>>>>>>>>> send
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> another patch.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for all the work on this everyone!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:29 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sure, that sounds good.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 12:11 PM, Wei ZHOU <
> >>>>>>>> ustcweizhou@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Mike,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It looks the two feature do not have many conflicts in
> Java
> >>>>>>>> code,
> >>>>>>>>>>>>>> except
> >>>>>>>>>>>>>>>>> the cloudstack UI.
> >>>>>>>>>>>>>>>>> If you do not mind, I will merge disk_io_throttling
> branch
> >>>> into
> >>>>>>>>>>>> master
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> week, so that you can develop based on it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Wei
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2013/6/11 Mike Tutkowski <mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hey John,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The SolidFire patch does not depend on the object_store
> >>>>>>>> branch,
> >>>>>>>>>> but
> >>>>>>>>>>>> -
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>> Edison mentioned - it might be easier if we merge the
> >>>>>>>> SolidFire
> >>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>> the object_store branch before object_store goes into
> >>>> master.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'm not sure how the disk_io_throttling fits into this
> >> merge
> >>>>>>>>>>>> strategy.
> >>>>>>>>>>>>>>>>>> Perhaps Wei can chime in on that.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Jun 11, 2013 at 11:07 AM, John Burwell <
> >>>>>>>>>> jburwell@basho.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Mike,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We have a delicate merge dance to perform.  The
> >>>>>>>>>> disk_io_throttling,
> >>>>>>>>>>>>>>>>>>> solidfire, and object_store appear to have a number of
> >>>>>>>>>> overlapping
> >>>>>>>>>>>>>>>>>>> elements.  I understand the dependencies between the
> >>>> patches
> >>>>>>>> to
> >>>>>>>>>> be
> >>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>> follows:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>  object_store <- solidfire -> disk_io_throttling
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Am I correct that the device management aspects of
> >>>> SolidFire
> >>>>>>>> are
> >>>>>>>>>>>>>>>>> additive
> >>>>>>>>>>>>>>>>>>> to the object_store branch or there are circular
> >> dependency
> >>>>>>>>>> between
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> branches?  Once we understand the dependency graph, we
> >> can
> >>>>>>>>>>>> determine
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> best approach to land the changes in master.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Jun 10, 2013, at 11:10 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Also, if we are good with Edison merging my code into
> >> his
> >>>>>>>> branch
> >>>>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>> going into master, I am good with that.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We can remove the StoragePoolType.Dynamic code after
> his
> >>>>>>>> merge
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>> deal with Burst IOPS then, as well.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 9:08 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Let me make sure I follow where we're going here:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1) There should be NO references to hypervisor code
> in
> >>>> the
> >>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>> plug-ins code (this includes the default storage
> >> plug-in,
> >>>>>>>> which
> >>>>>>>>>>>>>>>>>>> currently
> >>>>>>>>>>>>>>>>>>>>> sends several commands to the hypervisor in use
> >> (although
> >>>>>>>> it
> >>>>>>>>>> does
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> know
> >>>>>>>>>>>>>>>>>>>>> which hypervisor (XenServer, ESX(i), etc.) is
> actually
> >> in
> >>>>>>>> use))
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 2) managed=true or managed=false can be placed in the
> >> url
> >>>>>>>> field
> >>>>>>>>>>>> (if
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> present, we default to false). This info is stored in
> >> the
> >>>>>>>>>>>>>>>>>>>>> storage_pool_details table.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 3) When the "attach" command is sent to the
> hypervisor
> >> in
> >>>>>>>>>>>>>>>>> question, we
> >>>>>>>>>>>>>>>>>>>>> pass the managed property along (this takes the place
> >> of
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> StoragePoolType.Dynamic check).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 4) execute(AttachVolumeCommand) in the hypervisor
> >> checks
> >>>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> managed
> >>>>>>>>>>>>>>>>>>>>> property. If true for an attach, the necessary
> >> hypervisor
> >>>>>>>> data
> >>>>>>>>>>>>>>>>>>> structure is
> >>>>>>>>>>>>>>>>>>>>> created and the rest of the attach command executes
> to
> >>>>>>>> attach
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> volume.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 5) When execute(AttachVolumeCommand) is invoked to
> >>>> detach a
> >>>>>>>>>>>> volume,
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> same check is made. If managed, the hypervisor data
> >>>>>>>> structure
> >>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> removed.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 6) I do not see an clear way to support Burst IOPS in
> >> 4.2
> >>>>>>>>>> unless
> >>>>>>>>>>>>>>>>> it is
> >>>>>>>>>>>>>>>>>>>>> stored in the volumes and disk_offerings table. If we
> >>>> have
> >>>>>>>> some
> >>>>>>>>>>>>>>>>> idea,
> >>>>>>>>>>>>>>>>>>>>> that'd be cool.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Mon, Jun 10, 2013 at 8:58 PM, Mike Tutkowski <
> >>>>>>>>>>>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> "+1 -- Burst IOPS can be implemented while avoiding
> >>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>> attributes.  I always wondered about the details
> >> field.
> >>>> I
> >>>>>>>>>> think
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>> beef up the description in the documentation
> regarding
> >>>> the
> >>>>>>>>>>>>>>>>> expected
> >>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>> of the field.  In 4.1, I noticed that the details
> are
> >>>> not
> >>>>>>>>>>>>>>>>> returned on
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> createStoratePool updateStoragePool, or
> >> listStoragePool
> >>>>>>>>>>>> response.
> >>>>>>>>>>>>>>>>>> Why
> >>>>>>>>>>>>>>>>>>>>>> don't we return it?  It seems like it would be
> useful
> >>>> for
> >>>>>>>>>>>> clients
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> able to inspect the contents of the details field."
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Not sure how this would work storing Burst IOPS
> here.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Burst IOPS need to be variable on a Disk
> >>>> Offering-by-Disk
> >>>>>>>>>>>> Offering
> >>>>>>>>>>>>>>>>>>>>>> basis. For each Disk Offering created, you have to
> be
> >>>>>>>> able to
> >>>>>>>>>>>>>>>>>> associate
> >>>>>>>>>>>>>>>>>>>>>> unique Burst IOPS. There is a disk_offering_details
> >>>> table.
> >>>>>>>>>> Maybe
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>> go there?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I'm also not sure how you would accept the Burst
> IOPS
> >> in
> >>>>>>>> the
> >>>>>>>>>> GUI
> >>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>> not stored like the Min and Max fields are in the
> DB.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>>>> cloud<
> >> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>>>>> cloud<
> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>>> Advancing the way the world uses the cloud<
> >>>>>>>>>>>>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>>>> *™*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>>>> *™*
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> *Mike Tutkowski*
> >>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>>>> o: 303.746.7302
> >>>>>>>>>>> Advancing the way the world uses the
> >>>>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>>>> *™*
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Mike Tutkowski*
> >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>> o: 303.746.7302
> >>>>>>>>> Advancing the way the world uses the
> >>>>>>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>>>>>> *™*
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> *Mike Tutkowski*
> >>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>> o: 303.746.7302
> >>>>>>> Advancing the way the world uses the cloud<
> >>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>>> *™*
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud<
> >>>> http://solidfire.com/solution/overview/?video=play>
> >>>>>> *™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the
> >>>>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>>>> *™*
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the
> >>> cloud<http://solidfire.com/solution/overview/?video=play>
> >>> *™*
> >>
> >>
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud<http://solidfire.com/solution/overview/?video=play>
> > *™*
>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message