cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhinandan Prateek <abhinandan.prat...@shapeblue.com>
Subject Re: Next ACS release?
Date Thu, 23 Apr 2015 10:44:11 GMT
A smaller release cycle benefits individual companies running their own cloud. Basically they
are looking for the next versions that have features that they need or are incrementally moving
to new releases smoothly as they know what they need, the test environment and deployment
is with them and can quickly figure out if a release is worth moving to or not.

Companies providing services need to worry about all the versions out there that their customers
are using. In general with many versions around you need to support and patch more and more
versions as time goes on. The QA needs to be more extensive here as the real deployment is
somewhere else. More releases in use create bigger overhead.

The release timelines suggested are from 2 weeks to 6 months !   We can target for a smaller
release cycle but I don’t think we are there yet to have a two week release cycle.
For a smaller release cycle the RM needs to ensure that very specific code gets in. The test
coverage is measured and is good so that it almost eliminates the possibility of regression.
To decrease release cycle companies should collaborate more on QA, like they can divide the
QA work and also share the QA automation results.

Some of this is already happening like the infra that SBP, Daan and others have put in place,
I accept my ignorance of some of these things that are already there.

We can target for a major release very six month and can have incremental fortnightly releases
with control on patches and fixes that get in.

-abhi


> On 23-Apr-2015, at 3:26 pm, Ian Southam <ISoutham@schubergphilis.com> wrote:
>
> At SBP, we are working on two environments for testing.  Primarily focussed on Xen and
KVM with Nicira and without Nicira.
>
> One will focus more on building and extending the current integration tests and making
sure they are running on real hardware in real life situations.  The other we are planning
to be more “chaos monkey” in operation.  So genuinely testing how the system reacts to
hypervisors crashing.  Loss of network connectivity, VR failures/failovers and so on.
>
> I know other people in the community are doing similar things.  By having enough such
systems that together have a high coverage of all the various configuration and hardware combinations
out there, we should be able to really get to a much shorter delivery cycle with quite high
levels of confidence about quality.
>
> We are not there yet but, I am absolutely convinced that aiming for a 2 week release
cycle is the way to go.
>
> —
> Grts!
> Ian
>
>> On 23 Apr 2015, at 10:38, Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
wrote:
>>
>> On automated QA front following is available:
>>
>> 1. Before pushing in a feature a dev can run simulator based tests that will basically
test various functionality that does not depend on the type of Hypervisor.
>> (https://cwiki.apache.org/confluence/display/CLOUDSTACK/Validating+check-ins+for+your+local+changes%2C+using+Simulator.)
>> The test suite are located in testing/integration/smoke folder.
>> The travis system runs most of the test in this folder.
>>
>>
>> 2. Then there are tests that will require real hardware to run most of these are
in testing/integration/component.
>>
>>
>> Basically there are two kind of test cases not strictly classified as per above directory
structure - Ones that have required_hardware set as “false” and “others” that have
this as “true”.
>> The one with required_hardware is false can run on simulator but for the others you
need a real Hypervisor based environment.
>>
>> I have been able to run a lot of tests both with hardware and simulator. The problem
I faced is scattered documentation. Missing description of a model deployment; say for a particular
Hypervisor that will allow a dev to run the provisioning tests.
>>
>> In all there is a huge scope for improvement.
>>
>>
>> -abhi
>>
>>
>>> On 23-Apr-2015, at 1:02 am, Remi Bergsma <remi@remi.nl> wrote:
>>>
>>> Hi,
>>>
>>> The '2 week cycle' was intended as something to work towards, like a
>>> mission. The nice thing is that it immediately shows that we cannot achieve
>>> such a thing if we keep our current method of (semi)manual testing, as you
>>> said. Totally agree.
>>>
>>> That's why we need to improve on our automated functional testing. And that
>>> will of course take time and effort. As I don't currently have a clear
>>> overview of what is already available, I'll try to get that first and work
>>> from there. I spoke to several people recently and most seem to do testing
>>> on their own way (with sometimes cool automatic stuff going on!). It'd be
>>> interesting to bring that together and share.
>>> I think improving this is important, so I'll try to spend as much time on
>>> this as possible.
>>>
>>> However, the tread started with comments on 4.5. Let's try to get it stable
>>> and deliver 4.5.1. What is still to be done here? Can we share the load
>>> somehow to fasten it?
>>>
>>> Regards,
>>> Remi
>>>
>>> 2015-04-22 20:13 GMT+02:00 Paul Angus <paul.angus@shapeblue.com>:
>>>
>>>> I fully support the idea of a stable master with an automated CD process
>>>> to protect against regressions.
>>>>
>>>> ....However, we obviously don't currently have fantastic integration
>>>> testing otherwise we wouldn't have relied on 'people' to pick up the
>>>> release blockers recently.  A 2 week release cycle IMHO is way too frequent
>>>> to expect 'people' to be carrying out integration testing.
>>>>
>>>> Maybe 1 month is a workable compromise until the integration testing is of
>>>> a coverage and standard that can give real confidence.
>>>>
>>>>
>>>>
>>>> I'm also going to compile a list of people who vote "+1 - it works on my
>>>> laptop" and devise a Guinness related punishment for any of them that show
>>>> up at the CloudStack day in Dublin.
>>>>
>>>> Regards
>>>>
>>>> Paul Angus
>>>> Cloud Architect
>>>> S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
>>>> paul.angus@shapeblue.com
>>>>
>>>> -----Original Message-----
>>>> From: Remi Bergsma [mailto:remi@remi.nl]
>>>> Sent: 22 April 2015 10:25
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: Next ACS release?
>>>>
>>>> I'd be happy to help here as well. Last week in Austin, we also discussed
>>>> this topic a couple of times. I do agree shorter release cycles are better.
>>>>
>>>> In my opinion, the first thing to improve is the minor releases in both the
>>>> 4.4 and 4.5 branches. If we speed those up to let's say once every 2-weeks
>>>> we will be able to do the next minor release with less effort and users can
>>>> choose to either wait to start using 4.5 or start now and upgrade when the
>>>> next minor release is available. This would have helped in getting 4.5 out
>>>> on time and quickly fixing issues after the initial release. Also, it will
>>>> save time which we can invest in getting the next release out on time, etc.
>>>>
>>>> The common thing here is we need more automated testing, preferably
>>>> functional testing in addition to unit testing. There might also be other
>>>> steps that we do manually now that can be automated. I'll try to understand
>>>> the current process first and then come up with a proposal to improve which
>>>> we can discuss.
>>>>
>>>> Regards,
>>>> Remi
>>>>
>>>> 2015-04-22 10:56 GMT+02:00 Erik Weber <terbolous@gmail.com>:
>>>>
>>>>> On Wed, Apr 22, 2015 at 10:46 AM, Daan Hoogland
>>>>> <daan.hoogland@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Apr 22, 2015 at 10:29 AM, Erik Weber <terbolous@gmail.com>
>>>>> wrote:
>>>>>>> On Wed, Apr 22, 2015 at 9:49 AM, Stephen Turner <
>>>>>> Stephen.Turner@citrix.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have to admit I'm a bit sceptical because when we did have
a
>>>>>> four-month
>>>>>>>> release cycle, we never seemed to manage to meet it. Personally
I
>>>>> think
>>>>>>>> six-monthly might be easier.
>>>>>>>>
>>>>>>>> Having said that, part of the problem was the long close-down
>>>>>>>> period
>>>>>> where
>>>>>>>> we kept finding critical bugs, so more automated testing
might
>>>>>>>> help to shorten the cycle.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> - Shorter cycle means that it's less of a problem to miss a
>>>>>>> deadline, meaning you can spend more time on QA.
>>>>>>> - Longer cycle means that it could be critical to meet the
>>>>>>> deadline, meaning you might end up doing less QA while stressing
>>>>>>> to get your
>>>>>> feature
>>>>>>> in.
>>>>>>>
>>>>>>> My $.002
>>>>>>
>>>>>>
>>>>> Yes, Eric and the amount of qa needed is less as well. Bare with me
>>>>>> for a little rant here.
>>>>>>
>>>>>> When less new features are introduced less new interdependencies
are
>>>>>> introduced. I could try and expand on this but people make phd on
>>>>>> the subject so that would be presumptuous of me.
>>>>>>
>>>>>> un-educated guess is (m + n)! - m!
>>>>>> where m is old features and n is new features
>>>>>>
>>>>>> example:
>>>>>> 1 old feature + 1 new feature mean 1 check to see if they (still)
>>>>>> work
>>>>>> 1 old feature + 2 new features means 3 checks to see if they all
>>>>>> work
>>>>>> 2 + 2 = 6 of which only 1 is old and should be fine
>>>>>> 2 + 3 = 10, see the n! progressing ;)
>>>>>>
>>>>>>
>>>>>> this is an over simplification as the complexity of features comes
>>>>>> in play as well. I make them out for being function points here.
>>>>>> those are fables of course.
>>>>>>
>>>>>> thanks for baring that.
>>>>>>
>>>>>>
>>>>> I'm all with you on this Daan.
>>>>> Just for the record, my notion of QA was meant at the feature
>>>>> developer, ie. that they could use more time on test coverage etc.
>>>>> without having to worry that much about feature freeze dates.
>>>>>
>>>>> --
>>>>> Erik
>>>>>
>>>> Find out more about ShapeBlue and our range of CloudStack related services
>>>>
>>>> IaaS Cloud Design & Build<
>>>> http://shapeblue.com/iaas-cloud-design-and-build//>
>>>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>>>> CloudStack Software Engineering<
>>>> http://shapeblue.com/cloudstack-software-engineering/>
>>>> CloudStack Infrastructure Support<
>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>> CloudStack Bootcamp Training Courses<
>>>> http://shapeblue.com/cloudstack-training/>
>>>>
>>>> This email and any attachments to it may be confidential and are intended
>>>> solely for the use of the individual to whom it is addressed. Any views or
>>>> opinions expressed are solely those of the author and do not necessarily
>>>> represent those of Shape Blue Ltd or related companies. If you are not the
>>>> intended recipient of this email, you must neither take any action based
>>>> upon its contents, nor copy or show it to anyone. Please contact the sender
>>>> if you believe you have received this email in error. Shape Blue Ltd is a
>>>> company incorporated in England & Wales. ShapeBlue Services India LLP
is a
>>>> company incorporated in India and is operated under license from Shape Blue
>>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
>>>> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
>>>> a company registered by The Republic of South Africa and is traded under
>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related services
>>
>> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are intended solely
for the use of the individual to whom it is addressed. Any views or opinions expressed are
solely those of the author and do not necessarily represent those of Shape Blue Ltd or related
companies. If you are not the intended recipient of this email, you must neither take any
action based upon its contents, nor copy or show it to anyone. Please contact the sender if
you believe you have received this email in error. Shape Blue Ltd is a company incorporated
in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and
is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty
Ltd is a company registered by The Republic of South Africa and is traded under license from
Shape Blue Ltd. ShapeBlue is a registered trademark.
>

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.
Mime
View raw message