airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amila Jayasekara <thejaka.am...@gmail.com>
Subject Re: [DISCUSS] Release Methodology
Date Tue, 17 Dec 2013 13:14:00 GMT
This is nice !

-AJ


On Tue, Dec 17, 2013 at 3:41 AM, Saminda Wijeratne <samindaw@gmail.com>wrote:

> To see how it might go I created a simple spreadsheet [1] to record test
> results. "0" for untested, positve value for passed tests and negative
> value for failed tests.
>
> I realized it would be overwhelming for a single developer to carryout all
> the tests so I think its easier to just go on with the usual tests we do
> and mark them in the spreadsheet what we covered and later RM (or someone)
> can figureout a way to carryout tests which was not covered by anyone.
>
> Let me know if anyone needs edit privileges to the spreadsheet.
>
> Saminda
>
> 1.
> https://docs.google.com/spreadsheet/ccc?key=0AqYI3-ZrFz-EdFI0S3htaWJ1MDV4RE1WM19Ga0lhbEE&usp=sharing#gid=0
>
>
> On Mon, Dec 16, 2013 at 1:12 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> +1, please.
>>
>> - Thejaka Amila
>>
>>
>> On Mon, Dec 16, 2013 at 2:05 PM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> On Dec 16, 2013, at 1:51 PM, Saminda Wijeratne <samindaw@gmail.com>
>>> wrote:
>>>
>>> > I was thinking of an actual checklist where we can check-off/vote-off
>>> once each test is done. Perhaps we can start with a simple spreadsheet with
>>> the Tests specified by Raman added.
>>>
>>> + 1. Here is an example from Rave. Template for Quality Assurance [1]
>>> and an example [2].
>>>
>>> Bottom line, for atleast few days during the release process, we all
>>> should become the QA Team.
>>>
>>> Currently, we are doing scripted testing like 5, 10 minute tutorials and
>>> grid job submissions and lot of code still does not get touched. As an
>>> example, provenance aware search became nonfunctional and until Sanjaya
>>> pointed it out, we did not notice it. It will be useful, if randomly (or by
>>> co-ordination) we all test an RC against various features and then post
>>> them to DISCUSS thread. Otherwise, the releases just become pointing to a
>>> tag. We need to move from releases being a formality to every release
>>> robusting the code. We have so much active development and if we turn some
>>> energy to testing and bug fixing, I think our users will be happy with the
>>> outcome.
>>>
>>> Suresh
>>> [1] - http://wiki.apache.org/rave/ReleaseManagement/QualityAssurance
>>> [2] -
>>> http://wiki.apache.org/rave/ReleaseManagement/ReleaseSchedule/VerificationResults-0.11
>>> >
>>> >
>>> > On Mon, Dec 16, 2013 at 10:24 AM, Chathuri Wimalasena <
>>> kamalasini@gmail.com> wrote:
>>> > There is a general checklist added by Raman [1], which covers basic
>>> functionalities.
>>> >
>>> > Thanks..
>>> > Chathuri
>>> >
>>> > [1]
>>> https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Release+Testing
>>> >
>>> >
>>> > On Mon, Dec 16, 2013 at 12:56 PM, Saminda Wijeratne <
>>> samindaw@gmail.com> wrote:
>>> >
>>> >
>>> >
>>> > On Mon, Dec 16, 2013 at 9:28 AM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>> > Thanks Amila for weighing in. Comments inline:
>>> >
>>> > On Dec 16, 2013, at 11:29 AM, Amila Jayasekara <
>>> thejaka.amila@gmail.com> wrote:
>>> >
>>> > > Hi Suresh,
>>> > >
>>> > > I have some comments inline.
>>> > >
>>> > >
>>> > > On Mon, Dec 16, 2013 at 10:53 AM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>> > > Hi All,
>>> > >
>>> > > This is a very good question. Lets discuss these options so we are
>>> consistent across releases.
>>> > >
>>> > > If we look at the way we are doing releases, we are calling a
>>> feature freeze and code freeze and cutting a release. Most of the time, our
>>> build is broken. Jenkins   statistics for Airavata is not looking good at
>>> all [1].
>>> > >
>>> > > There is something wrong with the Jenkins configurations. I tried to
>>> figure out sometime back I was unable to do so. Even though builds are
>>> successful in our local machines they are failing intermittently in Jenkins.
>>> > >
>>> > > We are barely fixing the build a day before the release, putting out
>>> an RC and testing on it and releasing it in a quick succession.
>>> > >
>>> > > This is not entirely true. For the past few months I only
>>> experienced one or two build breaks (maybe less). I build couple of times
>>> per week. I believe usually build is stable and with integration tests
>>> passing, we always get a workable version. I know its not a good practice
>>> not to rely on the build server. But commiters have personal discipline to
>>> keep the build stable. Nevertheless we must fix Jenkins configuration issue.
>>> >
>>> > May be we should put focus on Jenkins configuration? Any volunteers?
>>> >
>>> > >
>>> > > As we are seeing on user lists, we have users upgrading with every
>>> release. I think we should increase the release quality.
>>> > >
>>> > > +1 for this.
>>> > >
>>> > > I would vote for atleast 3 RC’s per release. If we are not finding
>>> issues in first RC, I would say, either the software has magically become
>>> too too good or we are not doing through testing. I suspect the later.
>>> > How about we keep a checklist of release tests? I know we already send
>>> a mail on dev on what needs to be tested for each RC, but I need that is
>>> too abstract. For core developers of Airavata I think there should be test
>>> cases predefined (a test document if you may). Since we have several core
>>> developers in the list we can atleast decide upon what must be tested and
>>> make sure that each test case is covered by atleast one developer for a RC.
>>> > >
>>> > > I guess you mentioned this under assumption that build is not stable.
>>> >
>>> > Half of my assumption is on Jenkins, so if builds are ok and Jenkins
>>> is thinking wrong, then we can alleviate it by fixing it.
>>> >
>>> > > I will propose the following, please counter it and lets agree on a
>>> process:
>>> > >
>>> > > * Lets post a RC1 as is (which means it will have a snapshot). This
>>> pack, we should all test as much as possible, so its more of a test
>>> candidate then a release candidate. If it helps, we can use the name TC1. I
>>> am not particular on the naming but trying to emphasize the need for having
>>> atleast more RC's per release.
>>> > >
>>> > > I am not sure whether we really need a TC. The release manager
>>> should be doing some verifications on the RC before putting it out.
>>> Therefore it should be a RC. Anyhow i am fine having TC concept and trying
>>> it out.
>>> >
>>> > We probably should stick to RC, but I think the onus should not be on
>>> the RM to test it. They should coordinate and mobilize every one to do the
>>> testing including doing a testing bit more than others. But my point is, we
>>> should test and the only way to do that is to put a series of RC’s and have
>>> focused testing.
>>> > A TC should be something internal IMO. But when we are going for a
>>> release it should be alpha, beta and then RC releases. I think it need not
>>> be mandatory for the RMs to do pre-evaluation of the builds other than
>>> making sure all the unit tests and integration tests pass. Once an RC is
>>> confirmed of release quality I think we can follow the actual release cycle
>>> from the trunk itself with since its in a code freeze anyway.
>>> >
>>> > Suresh
>>> >
>>> > >
>>> > > What we really need is set of verifiable test cases.
>>> > >
>>> > > Thank you
>>> > > Regards
>>> > > Amila
>>> > >
>>> > >
>>> > > * If we do not expose significant issues in RC/TC 1 then we proceed
>>> with RC2 which will follow the proper release process. But if we have a
>>> reasonable issues bought out, we need a RC2/TC2 also without following the
>>> release process.
>>> > >
>>> > > * The key thing I am proposing is, we keep doing RC/TC’s until we
>>> all are sure the quality is good enough with documented known issues. When
>>> we are sure, then we proceed to have RC with proper release process.
>>> > >
>>> > > So this will mean more testing and twice (or more) the times every
>>> one has to test, but I think it is worth it. This might also get over the 6
>>> week release cycle, but I think we need to trade for some quality releases
>>> as we march towards 1.0.
>>> > >
>>> > > Suresh
>>> > > [1] - https://builds.apache.org/job/Apache%20Airavata/
>>> > >
>>> > >
>>> > > On Dec 15, 2013, at 4:28 PM, Lahiru Gunathilake <glahiru@gmail.com>
>>> wrote:
>>> > >
>>> > > >
>>> > > > Hi Chathuri,
>>> > > >
>>> > > > I think having snapshot as the version in RC is wrong. Every RC
>>> has to be like a release and if it pass we just call a vote/discussion
>>> thread and do the release. If we do with snapshot  and if things go right,
>>> then have to change versions and test again. But we can do the release just
>>> by changing snapshot without testing but that wrong AFAIT.
>>> > > >
>>> > > > I remember doing this mistake in earlier release with RC1 build.
I
>>> think we can stick to the release management instructions in
>>> airavata.org.
>>> > > >
>>> > > > Regards
>>> > > > Lahiru
>>> > > >
>>> > > >
>>> > > > On Fri, Dec 13, 2013 at 3:43 PM, Chathuri Wimalasena <
>>> kamalasini@gmail.com> wrote:
>>> > > > Hi All,
>>> > > >
>>> > > > Airavata 0.11 RC1[1] is ready for testing.
>>> > > >
>>> > > > Here are some pointers for testing
>>> > > >       • Verify the fixed issue for this release [2]
>>> > > >       • Verify the basic workflow composition/execution/monitoring
>>> scenarios from
>>> > > >       • Airavata 5 & 10 min tutorials [3],[4]
>>> > > >       • Verify airavata client samples
>>> > > >       • Verify the stability with derby & mysql backend
databases
>>> > > >       • Verify that the XBaya JNLP distribution works
>>> > > >       • Verify deploying Airavata server in a tomcat distribution
>>> > > > Please report any issues[5] if you encounter while testing. Thank
>>> you for your time in validating the release.
>>> > > >
>>> > > > Regards,
>>> > > > Chathuri (On behalf of Airavata PMC)
>>> > > >
>>> > > > [1] https://dist.apache.org/repos/dist/dev/airavata/0.11/RC1/
>>> > > > [2]
>>> https://issues.apache.org/jira/browse/AIRAVATA-278?jql=project%20%3D%20AIRAVATA%20AND%20fixVersion%20%3D%20%220.11%22%20ORDER%20BY%20status%20DESC%2C%20priority%20DESC
>>> > > > [3]
>>> http://airavata.apache.org/documentation/tutorials/airavata-in-5-minutes.html
>>> > > > [4]
>>> http://airavata.apache.org/documentation/tutorials/airavata-in-10-minutes.html
>>> > > > [5] https://issues.apache.org/jira/browse/AIRAVATA
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > System Analyst Programmer
>>> > > > PTI Lab
>>> > > > Indiana University
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>

Mime
View raw message