cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Tutkowski <mike.tutkow...@solidfire.com>
Subject Re: feature : changing volume properties dynamically in 4.5
Date Fri, 20 Jun 2014 05:07:06 GMT
I just wanted to update those who are interested in this thread about work
I've done on this over the past couple days.

This gist is that I've added a new method to the PrimaryDataStoreLifeCycle
interface that has the following signature (this code is not yet checked
in):

void updateCapacity(StoragePool storagePool, Long capacityBytes, Long
capacityIops);

This method can be invoked from StorageManagerImpl when the
updateStoragePool API is called.

This gives the storage plug-in that backs the primary storage in question
an opportunity to update the backend storage it represents, if that makes
sense to do in your particular case (for example, changing the size and/or
IOPS of a volume).

There is a related enhancement to the resizeVolume API that I plan to
tackle next week. That one will be particularly useful for managed storage
plug-ins.

Also, I have been collecting input on the generic key/value proposal here:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111

That may turn into a considerable amount of work. I was initially thinking
it could be fit into 4.5, but it might be 4.6.

Thanks for any feedback!


On Thu, Jun 12, 2014 at 11:09 PM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> As I think about this more, there are two situations we should cover:
>
> 1) Non-managed storage that has control over IOPS.
>
> When you invoke the createStoragePool API, you can pass in capacityIops.
>
> We should support modifying capacityIops via the updateStoragePool API.
>
> 2) Managed storage that has control over IOPS.
>
> In this environment, there is a 1:1 mapping between a SAN volume and a
> CloudStack volume.
>
> This is where we need to augment the resizeVolume API to accept - in a
> similar fashion to size - a new value for Min and/or Max IOPS.
>
> For example, a resizeVolume can be initiated by simply selecting a new
> Disk Offering.
>
> In this situation, the size and IOPS are part of the new Disk Offering
> (i.e. neither size nor IOPS can be marked as custom) and when the resize
> method of the storage plug-in is invoked, it will have a chance to change
> the size and/or IOPS.
>
> I would say we should perform a bit of analysis in the CloudStack volume
> logic to NOT stop resize from being invoked on the storage plug-in IF the
> volume size is the same, but the IOPS are different. This way the volume
> can be online as long as the user is only changing the IOPS of the volume.
>
> I also think it's only a problem for XenServer for the VDI to have its
> size changed dynamically.
>
> I plan to draw a flowchart for this soon. Once I do that I think it will
> be easier to talk in detail.
>
> Thanks!
>
>
> On Thu, Jun 12, 2014 at 12:59 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> From what I understand about the resizeVolume API, to change the size of
>> a given volume, you must either:
>>
>> 1) pass in a new Disk Offering (if the current Disk Offering the volume
>> uses does not allow for a custom size)
>>
>> or
>>
>> 2) pass in the ID of the volume and a new size (if the current Disk
>> Offering the volume uses does allow for a custom size)
>>
>> Either way, if you are shrinking the volume's size, you then have to pass
>> in 'true' for the 'shrinkok' parameter.
>>
>> One thing we should do is support this same concept with IOPS. At the
>> time being, both Min and Max IOPS can be custom (set by user) or non custom
>> (set by admin). This is a direct parallel to custom size or non-custom
>> size. If the user is using a non-custom IOPS setting and wants to switch to
>> a custom IOPS setting, he should be able to do so by switching to a Disk
>> Offering with custom IOPS. Of course we should support doing this while the
>> volume is attached.
>>
>> If arbitrary key/value pairs can be associated with Disk Offerings, then
>> you should be able to get the new key/value pairs by switching to a new
>> Disk Offering. We'd want to allow this to work with the volume in the
>> attached state, as well.
>>
>> Perhaps we should allow this all to happen online (volume attached)
>> UNLESS doing what we're about to do will change the size of the volume.
>> Then we can fail the OP and tell them to detach the volume and re-run the
>> OP.
>>
>> What are you thoughts on that?
>>
>> Also, I think volumeResize only works for data disks at the time being.
>>
>> In my mind, volumeResize is a bit of a misnomer now. We are really
>> allowing the user to resize their Disk Offering now in terms of not only
>> size, but IOPS, and even arbitrary key/value pairs. This is still all done
>> by selecting a new Disk Offering (or - if you have a custom size or custom
>> IOPS disk offering already - by passing in the ID of the volume and the new
>> size and/or IOPS).
>>
>> Let's brainstorm on this a bit and see which way makes sense to go.
>>
>> Thanks!
>>
>>
>> On Thu, Jun 12, 2014 at 9:48 AM, Punith S <punith.s@cloudbyte.com> wrote:
>>
>>> sure mike.
>>>
>>> and i have one question,
>>>
>>> which existing volume api are we gonna use for changing disk
>>> offerings(properties) dynamically ?
>>> since resize api will not allow unless the disk is detached !
>>>
>>> thanks.
>>>
>>>
>>> On Thu, Jun 12, 2014 at 11:37 AM, Mike Tutkowski <
>>> mike.tutkowski@solidfire.com> wrote:
>>>
>>>> Sounds good - let me give some thought as to how we should break up the
>>>> work.
>>>>
>>>> My GSoC student from Tunisia will be helping us, as well.
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 8:34 AM, Punith S <punith.s@cloudbyte.com>
>>>> wrote:
>>>>
>>>>> yes it sounds good, thanks for the proposal mike,
>>>>>
>>>>> yeah right now i have implemented prototype of my proposal, since its
>>>>> not generic we shall implement your proposal for 4.5.
>>>>> on the other hand, for 4.5 i'm supporting nfs protocol and resize
>>>>> feature for managed storage in xenserver, now trying to implement snapshot
>>>>> and support root disk for vm's.
>>>>> and yes if we can club together, i can start working on this proposal
>>>>> for data disk and get the prototype ready.
>>>>> what do you think ?
>>>>>
>>>>> thanks.
>>>>>
>>>>>
>>>>> On Wed, Jun 11, 2014 at 3:53 AM, Mike Tutkowski <
>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>>> I'll send out a [PROPOSAL] e-mail so others who may not be following
>>>>>> this e-mail thread have a better chance to comment on the feature.
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 10, 2014 at 2:58 PM, Mike Tutkowski <
>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>
>>>>>>> Here is that design document I was referring to, Punith:
>>>>>>>
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=42566111
>>>>>>>
>>>>>>> I've been working with a student in Tunisia who is participating
in
>>>>>>> Google Summer of Code (GSoC) (I'm his mentor).
>>>>>>>
>>>>>>> He'll be working on part of this as will I. (He is also working
on
>>>>>>> another related task not listed here.)
>>>>>>>
>>>>>>> Feel free to join us, if you have time available, as we can divide
>>>>>>> out coding and testing among the three of us.
>>>>>>>
>>>>>>> Talk to you later!
>>>>>>> Mike
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 10, 2014 at 10:18 AM, Mike Tutkowski <
>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>
>>>>>>>> I plan to draw up a design document surrounding generic key/value
>>>>>>>> pairs today.
>>>>>>>>
>>>>>>>> Perhaps you can take a look at it when you have time, Punith?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 10, 2014 at 10:06 AM, Mike Tutkowski <
>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Punith,
>>>>>>>>>
>>>>>>>>> Yeah, I hear you about the number of permutations involved.
>>>>>>>>>
>>>>>>>>> Traditionally Compute and Disk Offerings have been immutable.
It
>>>>>>>>> makes it easier from an accounting point of view for
chargeback and billing.
>>>>>>>>>
>>>>>>>>> You should definitely feel free to extend the CloudStack
API. I
>>>>>>>>> think NetApp did this for one of their storage features
in the recent past.
>>>>>>>>> This way vendor-specific capabilities can be more easily
offered without
>>>>>>>>> making it look like all vendors support those particular
features.
>>>>>>>>>
>>>>>>>>> I do not yet have any code in master related to generic
>>>>>>>>> keys/values. I'm still designing this.
>>>>>>>>>
>>>>>>>>> How does your schedule look for the 4.5 release? Do you
think you
>>>>>>>>> might have available cycles to help out with this generic
key/value
>>>>>>>>> implementation?
>>>>>>>>>
>>>>>>>>> Talk to you later!
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 10, 2014 at 7:09 AM, Punith S <punith.s@cloudbyte.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> hi mike,
>>>>>>>>>>
>>>>>>>>>> thanks for the reply, i like your approach which
is a very
>>>>>>>>>> generic way and also we only need to do a few changes
to the current
>>>>>>>>>> cloudstack,
>>>>>>>>>>
>>>>>>>>>> but on the other hand we are tying every property
of the vendor
>>>>>>>>>> to a disk offering through key/value pairs, since
we offer lot of
>>>>>>>>>> properties like i mentioned, this can create a lot
of permutation
>>>>>>>>>> combinations of disk offerings, for say if i need
to turn deduplication On
>>>>>>>>>> for a specific volume , should i have to create a
new disk offering having
>>>>>>>>>> current properties with deduplication On?
>>>>>>>>>>
>>>>>>>>>> is this approach already implemented in the current
master ?
>>>>>>>>>>
>>>>>>>>>> and also like you mentioned about exposing a new
api, is it okay
>>>>>>>>>> if i expose our own api in my util by extending the
PlugableService like in
>>>>>>>>>> network plugins ?
>>>>>>>>>>
>>>>>>>>>> thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 10, 2014 at 1:17 AM, Mike Tutkowski <
>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Allow me to follow this up with more detail (with
regards to
>>>>>>>>>>> what Chris and I talked about):
>>>>>>>>>>>
>>>>>>>>>>> As you are aware, today the way you associate
a Compute Offering
>>>>>>>>>>> (CO) or a Disk Offering (DO) with a Primary Storage
(PS) is via storage
>>>>>>>>>>> tagging.
>>>>>>>>>>>
>>>>>>>>>>> This has some benefits and drawbacks.
>>>>>>>>>>>
>>>>>>>>>>> One benefit is being able to have some level
of vendor
>>>>>>>>>>> independence from the point of view of the CO
or DO. For example, if the
>>>>>>>>>>> storage tag of a DO is "Fast", then this can
be satisfied by PS that
>>>>>>>>>>> describes itself as "Fast", regardless of vendor.
>>>>>>>>>>>
>>>>>>>>>>> A major drawback with the storage-tagging approach,
however, is
>>>>>>>>>>> that you are not easily able to leverage vendor-specific
features, which is
>>>>>>>>>>> often why you bought storage from the vendor
in question to begin with.
>>>>>>>>>>>
>>>>>>>>>>> Ideally we do not want to add each vendor's features
into the
>>>>>>>>>>> system as properties that can be seen by the
admin regardless of whether or
>>>>>>>>>>> not the underlying storage he's actually using
supports the feature in
>>>>>>>>>>> question.
>>>>>>>>>>>
>>>>>>>>>>> This coarse approach, however, was sort of business
as usual
>>>>>>>>>>> when I started in with CloudStack 1.5 years ago.
>>>>>>>>>>>
>>>>>>>>>>> That being the case, when I added QoS options
to CS, I did so in
>>>>>>>>>>> a way where the admin would see Min IOPS and
Max IOPS options regardless of
>>>>>>>>>>> whether or not his storage actually supported
those controls (to mitigate
>>>>>>>>>>> this a bit in the GUI, the admin has to explicitly
select "Storage QoS"
>>>>>>>>>>> from a combobox).
>>>>>>>>>>>
>>>>>>>>>>> We leverage the same use model with Hypervisor
QoS: The admin
>>>>>>>>>>> sees these options regardless of whether or not
they actually apply on the
>>>>>>>>>>> hypervisor where the VM gets deployed.
>>>>>>>>>>>
>>>>>>>>>>> Going forward, we want to implement a more fine-grain
and
>>>>>>>>>>> generic approach.
>>>>>>>>>>>
>>>>>>>>>>> We would like to have a storage provider field
for the CO and DO
>>>>>>>>>>> windows (this equates to the name of one and
only one storage provider). If
>>>>>>>>>>> the admin inputs a specific storage provider
and does not use the storage
>>>>>>>>>>> tags field, he can enter in an arbitrary number
of key/value pairs in
>>>>>>>>>>> another text field (perhaps we would provide
a nice entry dialog to make
>>>>>>>>>>> this easier in the GUI). These key value pairs
can be passed into the
>>>>>>>>>>> storage driver when it's asked to create (or
update) a volume and the
>>>>>>>>>>> storage driver can decide what each and every
key/value pair means.
>>>>>>>>>>>
>>>>>>>>>>> What do you think about this approach?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 9, 2014 at 1:14 PM, Mike Tutkowski
<
>>>>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Punith,
>>>>>>>>>>>>
>>>>>>>>>>>> This kind of a feature is something Chris
Suich and I discussed
>>>>>>>>>>>> a while back.
>>>>>>>>>>>>
>>>>>>>>>>>> We talked about leveraging arbitrary key/value
pairs to make
>>>>>>>>>>>> this happen (OpenStack does something similar).
The key/value pairs would
>>>>>>>>>>>> be vendor specific.
>>>>>>>>>>>>
>>>>>>>>>>>> If we take a key/value approach, we might
be able to make this
>>>>>>>>>>>> all work the way things work today when the
user wants to change an
>>>>>>>>>>>> existing Compute Offering and/or Disk Offering.
>>>>>>>>>>>>
>>>>>>>>>>>> For example, the user would pick a new Compute
Offering (with
>>>>>>>>>>>> presumably has different key/value pairs)
and CloudStack could inform the
>>>>>>>>>>>> applicable storage provider, who could update
the volume in question.
>>>>>>>>>>>>
>>>>>>>>>>>> This way we don't need to introduce a new
API command and the
>>>>>>>>>>>> use model for the user doesn't really change.
>>>>>>>>>>>>
>>>>>>>>>>>> What are you thoughts on this?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 9, 2014 at 8:08 AM, Punith S
<
>>>>>>>>>>>> punith.s@cloudbyte.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> hi guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> since most of the third party storage
providers have been
>>>>>>>>>>>>> implementing 1:1 mapping(managed storage)
between a volume(dataset) and a
>>>>>>>>>>>>> vm disk(vdi/vmdk) for guaranteeing the
Qos, i would like to propose a new
>>>>>>>>>>>>> feature to dynamically change the volume
properties supported by storage
>>>>>>>>>>>>> vendors such as IOPS, Deduplication,
Compression, Grace, Syncronization,
>>>>>>>>>>>>> Latency etc, depending on properties
and features supported by respective
>>>>>>>>>>>>> storage vendors. hence providing more
flexibility for users.
>>>>>>>>>>>>>
>>>>>>>>>>>>> in case of using default cloudstack storage
provider, we can
>>>>>>>>>>>>> change the properties of the vdi/vmdk
files apart from resizing the
>>>>>>>>>>>>> volume(vdi/vmdk).
>>>>>>>>>>>>>
>>>>>>>>>>>>> changes in management server include,
>>>>>>>>>>>>>
>>>>>>>>>>>>> new async web api ChangeVolumePropertiesCmd,
>>>>>>>>>>>>> new method in VolumeApiService for vo
and dao validation
>>>>>>>>>>>>> implementations.
>>>>>>>>>>>>> new method in VolumeServiceManager for
supporting callback and
>>>>>>>>>>>>> calling the respective storage provider
driver's implementation.
>>>>>>>>>>>>> new method in PrimaryDataStoreDriver
interface for
>>>>>>>>>>>>> implementing respective features according
to their storage product.
>>>>>>>>>>>>>
>>>>>>>>>>>>> changes in UI include,
>>>>>>>>>>>>> new changing volume properties widget
in volume section,
>>>>>>>>>>>>> showing different properties depending
upon listed storage providers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> any suggestions and feedbacks ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> punith s
>>>>>>>>>>>>> cloudbyte.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>>>  *Senior CloudStack Developer, SolidFire
Inc.*
>>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Mike Tutkowski*
>>>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>>>> o: 303.746.7302
>>>>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> regards,
>>>>>>>>>>
>>>>>>>>>> punith s
>>>>>>>>>> cloudbyte.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Mike Tutkowski*
>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>> o: 303.746.7302
>>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Mike Tutkowski*
>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>> o: 303.746.7302
>>>>>>> Advancing the way the world uses the cloud
>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Mike Tutkowski*
>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>> e: mike.tutkowski@solidfire.com
>>>>>> o: 303.746.7302
>>>>>> Advancing the way the world uses the cloud
>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> regards,
>>>>>
>>>>> punith s
>>>>> cloudbyte.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkowski@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud
>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>
>>>
>>>
>>>
>>> --
>>> regards,
>>>
>>> punith s
>>> cloudbyte.com
>>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message