cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raja Pullela <raja.pull...@accelerite.com>
Subject RE: [PROPOSAL] Minimum Viable CI Integration
Date Tue, 15 Mar 2016 08:59:47 GMT
Bharat, I will work Sanjeev on fixing the failed tests.  We can create a bug to track this.
 

Best,
Raja
-----Original Message-----
From: Remi Bergsma [mailto:RBergsma@schubergphilis.com] 
Sent: Tuesday, March 15, 2016 1:27 PM
To: dev@cloudstack.apache.org
Cc: Bharat Kumar <bharat.kumar@accelerite.com>
Subject: Re: [PROPOSAL] Minimum Viable CI Integration

Hi Bharat,

I’d suggest removing all tests that fail for some reason, not being an error in the PR.
This will result in a smaller set of tests that “always pass”. Once a test doesn’t pass
anymore, we know the PR has an issue. Then your CI system provides value.

Right now, there are so many failures that soon people don’t even care to look at it. Once
we have a stable set that works, we can start fixing tests and 1-by-1 add them to the mix.

Just my $0.02

Regards,
Remi



On 15/03/16 07:06, "Bharat Kumar" <bharat.kumar@accelerite.com> wrote:

>Hi Paul,
>
>I totally agree with you. For now we should ignore the tests which fail all the time while
we try to address them.
>
>Now that we have a CI in place we need go and fix issues with the tests.
>
>I have tried to identify a few of them and have  mentioned them in the known issues part
of the CI test result.
>
>Some of the test cases are not generic enough to run on different CIs. 
>For example the test_vpc_vpn.py hardcodes the test data in the test itself, which is not
a good thing. Generally this kind of data is added the the test_data.py file and is reused
.
>This gives us a chance to modify it to suit the needs of the a particular CI.
>
>So for fixing these issues we need test data to identify the issue. Each of the CI run
gives these details, the test run info, test results and the management server logs( via the
drop box link in the report).
>With this info any one in the community can access this data and start fixing them. This
way we can progressively track and improve the health of the release.
>
>So i think we should continue posting these results, so that people can analyse the result
and come back with inputs related to incorrect test cases.
>
>—Thanks
>Bharat.
>
>
>On 15-Mar-2016, at 1:24 AM, Paul Angus <paul.angus@shapeblue.com<mailto:paul.angus@shapeblue.com>>
wrote:
>
>Cool. Thanks Will.
>
>@Bharat - the work that you're trying to do is great, but until the Marvin tests have
been gone through and 'fixed' the information being put into PRs is misleading.
>
>No one knows why these failures are occurring, and it's very likely giving both false
positives and false negatives.
>You should probably hold off altogether or isolate the good tests and only run them.
>
>I plan to set up a wiki page (or two) with all of the tests listed, their purpose and
other info such as if there are known to work (or not) when they were last checked. You're
welcome to make a start on it if you wish.
>
>For example (false negative):
>------------------------------
>nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM 
>-a tags=advanced,required_hardware=true 
>integration/component/test_acl_isolatednetwork.py
>2016-01-30 07:36:30,488 - DEBUG - Parsing Test data successful
>2016-01-30 07:36:30,488 - DEBUG - Payload: {'account': 'admin', 
>'command': 'listUsers', 'response': 'json'}
>2016-01-30 07:36:30,488 - DEBUG - ========Sending GET Cmd : 
>listUsers=======
>2016-01-30 07:36:30,521 - DEBUG - Response : [{username : u'admin', 
>account : u'admin', domainid : u'4895ddbe-bfac-11e5-be38-06a910012727', 
>firstname : u'admin', created : u'2016-01-20T19:33:19+0000', lastname : 
>u'cloud', apikey : 
>u'oiD-zcjZw7dsPq8yrjarls3sCtHV0E8FQYRpNXlmEx0ssJ1fKFuCySYL4vLHVmlLA5E3P
>F7_hSnTND_eSRyK_A', domain : u'ROOT', secretkey : 
>u'gLz6lDVFch6ayBAB-8SMZ-RPiOgJTEVJmNgnAMdv36O6eMUKUN6R-Q-3ZtYJUH84jye-5
>q6Qpxf5WOtSuG8WtA', iscallerchilddomain : False, state : u'enabled', 
>accounttype : 1, id : u'a8b21230-bfac-11e5-be38-06a910012727', 
>isdefault : True, accountid : u'a8b2063c-bfac-11e5-be38-06a910012727'}]
>2016-01-30 07:36:30,521 - DEBUG - ==== Test Client Creation Successful 
>====
>-----------------------------------
>
>Nothing actually happens ?!
>Same with test_acl_listvm.py
>
>Example(false positive-failure is in marvin test code):
>-------------------------------------------------------------
>nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM 
>-a tags=advanced,required_hardware=true 
>/tmp/integration/component/test_browse_templates.py
>2016-01-30 19:52:26,160 - CRITICAL - EXCEPTION: None: ['Traceback (most 
>recent call last):\n', ' File 
>"/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run\n 
>self.setUp()\n', ' File 
>"/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp\n 
>self.setupContext(ancestor)\n', ' File 
>"/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in 
>setupContext\n try_run(context, names)\n', ' File 
>"/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n 
>return func()\n', ' File 
>"/tmp/integration/component/test_browse_templates.py", line 84, in 
>setUpClass\n 
>cls.uploadurl=cls.testdata["configurableData"]["browser_upload_template
>"][cls.uploadtemplateformat]["url"]\n', "KeyError: 
>'browser_upload_template'\n"]
>
>
>
>
>
><http://www.shapeblue.com/>
>Paul Angus
>VP Technology   ,       ShapeBlue
>
>
>d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
    |      m:      +44 7711 418784<tel:+44%207711%20418784>
>
>e:      paul.angus@shapeblue.com | t: @cloudyangus<mailto:paul.angus@shapeblue.com%20|%20t:%20@cloudyangus>
     |      w:      www.shapeblue.com<http://www.shapeblue.com/>
>
>a:      53 Chandos Place, Covent Garden London WC2N 4HS UK
>
>
>
>
>
>Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India
LLP is a company incorporated in India and is operated under license from Shape Blue Ltd.
Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under
license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered
trademark.
>This email and any attachments to it may be confidential and are intended solely for the
use of the individual to whom it is addressed. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error.
>
>
>
>
>-----Original Message-----
>From: Will Stevens [mailto:williamstevens@gmail.com]
>Sent: Monday, March 14, 2016 4:26 AM
>To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
>Subject: Re: [PROPOSAL] Minimum Viable CI Integration
>
>The latest version (0.2.0) of upr is now available...
>
>https://github.com/cloudops/upr
>
>All the details are in the README file of the repo, but the quick overview of the changes
include:
>
>- Added the ability to comment on a pull request (by PR number or 
>commit).
>- Implemented both S3 and Swift object storage backends for uploading 
>files (tested with AWS S3, Cloudian (S3) and Swift).
>- Allow for automatic expiration of uploaded files to clean up the 
>object store after a period of time (in both S3 and Swift).
>- Added the ability to override the default comment template with a 
>local file (look at the current template for details).
>
>Let me know if you guys have questions, comments or feedback on that...
>
>Hopefully the apache infra guys will get back to me tomorrow so we can better assess how
we need to move forward.
>
>Cheers,
>
>Will
>
>On Sun, Mar 13, 2016 at 7:11 PM, Will Stevens 
><williamstevens@gmail.com<mailto:williamstevens@gmail.com>>
>wrote:
>
>> Unfortunately this would only give us the ability to 'red 
>> light'/'green light' pull requests via a distributed CI environments.
>> The reason I limited the request to this permission is because 
>> anything they give us will be global to the entire apache org and 
>> this permission is restrictive enough that we can't do damage elsewhere.
>> There is a bit of history of them denying all requests of this 
>> manner, so I wanted to focus on our minimum viable use case, while limiting their
liability, to improve our chances.
>>
>> You can follow the action here:
>> https://issues.apache.org/jira/browse/INFRA-11429
>>
>> They have officially said no at this point, but I have asked them to 
>> reconsider and asked them to provide suggestions for solving our problem.
>>
>> If they say no again, they will be doing so with the full knowledge 
>> that the decision leaves us with our hands tied and we have exhausted 
>> our official options through them.
>>
>> If we end up in this situation, we will have to get creative and I 
>> suspect that will result in some activities moving outside the Apache 
>> repo. I have not gone there (yet) since I feel that will need to be 
>> voted on in an official thread of its own before we make a decision like that.
>>
>> Feedback and suggestions are very welcome.
>>
>> I am finalizing the 'comment' functionality in my tool 'upr' this 
>> weekend, so that could give us some limited options. I will try to 
>> push a release later tonight. It is all written and functional now, 
>> just need to clean up the documentation and examples.
>>
>Find out more about ShapeBlue and our range of CloudStack related services:
>IaaS Cloud Design & 
>Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – 
>rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | 
>CloudStack Software 
>Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>CloudStack Infrastructure 
>Support<http://shapeblue.com/cloudstack-infrastructure-support/> | 
>CloudStack Bootcamp Training 
>Courses<http://shapeblue.com/cloudstack-training/>
>
>
>
>
>DISCLAIMER
>==========
>This e-mail may contain privileged and confidential information which is the property
of Accelerite, a Persistent Systems business. It is intended only for the use of the individual
or entity to which it is addressed. If you are not the intended recipient, you are not authorized
to read, retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent
Systems business does not accept any liability for virus infected mails.



DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite,
a Persistent Systems business. It is intended only for the use of the individual or entity
to which it is addressed. If you are not the intended recipient, you are not authorized to
read, retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent
Systems business does not accept any liability for virus infected mails.
Mime
View raw message