cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remi Bergsma <RBerg...@schubergphilis.com>
Subject Re: [PROPOSAL] Minimum Viable CI Integration
Date Tue, 15 Mar 2016 11:06:35 GMT
Hi Bharat,

I’d say remove the tests that fail, to get a report with only tests that pass.

Regards,
Remi

From: Bharat Kumar <bharat.kumar@accelerite.com<mailto:bharat.kumar@accelerite.com>>
Date: Tuesday 15 March 2016 at 10:40
To: Remi Bergsma <RBergsma@schubergphilis.com<mailto:RBergsma@schubergphilis.com>>
Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>,
Bharat Kumar <bharat.kumar@accelerite.com<mailto:bharat.kumar@accelerite.com>>
Subject: Re: [PROPOSAL] Minimum Viable CI Integration

Hi Remi,

Thanks for the inputs.

I agree with you, May be i need to modify the CI report and not report if any of the known
issues fail.  We can still have the known_issues header to track what needs to be fixed.

Thanks,
Bharat.


On 15-Mar-2016, at 1:26 PM, Remi Bergsma <RBergsma@schubergphilis.com<mailto:RBergsma@schubergphilis.com>>
wrote:

Hi Bharat,

I’d suggest removing all tests that fail for some reason, not being an error in the PR.
This will result in a smaller set of tests that “always pass”. Once a test doesn’t pass
anymore, we know the PR has an issue. Then your CI system provides value.

Right now, there are so many failures that soon people don’t even care to look at it. Once
we have a stable set that works, we can start fixing tests and 1-by-1 add them to the mix.

Just my $0.02

Regards,
Remi



On 15/03/16 07:06, "Bharat Kumar" <bharat.kumar@accelerite.com<mailto:bharat.kumar@accelerite.com>>
wrote:

Hi Paul,

I totally agree with you. For now we should ignore the tests which fail all the time while
we try to address them.

Now that we have a CI in place we need go and fix issues with the tests.

I have tried to identify a few of them and have  mentioned them in the known issues part of
the CI test result.

Some of the test cases are not generic enough to run on different CIs. For example the test_vpc_vpn.py
hardcodes the test data in the test itself, which is not a good thing. Generally this kind
of data is added the the test_data.py file and is reused .
This gives us a chance to modify it to suit the needs of the a particular CI.

So for fixing these issues we need test data to identify the issue. Each of the CI run gives
these details, the test run info, test results and the management server logs( via the drop
box link in the report).
With this info any one in the community can access this data and start fixing them. This way
we can progressively track and improve the health of the release.

So i think we should continue posting these results, so that people can analyse the result
and come back with inputs related to incorrect test cases.

—Thanks
Bharat.


On 15-Mar-2016, at 1:24 AM, Paul Angus <paul.angus@shapeblue.com<mailto:paul.angus@shapeblue.com><mailto:paul.angus@shapeblue.com>>
wrote:

Cool. Thanks Will.

@Bharat - the work that you're trying to do is great, but until the Marvin tests have been
gone through and 'fixed' the information being put into PRs is misleading.

No one knows why these failures are occurring, and it's very likely giving both false positives
and false negatives.
You should probably hold off altogether or isolate the good tests and only run them.

I plan to set up a wiki page (or two) with all of the tests listed, their purpose and other
info such as if there are known to work (or not) when they were last checked. You're welcome
to make a start on it if you wish.

For example (false negative):
------------------------------
nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM -a tags=advanced,required_hardware=true
integration/component/test_acl_isolatednetwork.py
2016-01-30 07:36:30,488 - DEBUG - Parsing Test data successful
2016-01-30 07:36:30,488 - DEBUG - Payload: {'account': 'admin', 'command': 'listUsers', 'response':
'json'}
2016-01-30 07:36:30,488 - DEBUG - ========Sending GET Cmd : listUsers=======
2016-01-30 07:36:30,521 - DEBUG - Response : [{username : u'admin', account : u'admin', domainid
: u'4895ddbe-bfac-11e5-be38-06a910012727', firstname : u'admin', created : u'2016-01-20T19:33:19+0000',
lastname : u'cloud', apikey : u'oiD-zcjZw7dsPq8yrjarls3sCtHV0E8FQYRpNXlmEx0ssJ1fKFuCySYL4vLHVmlLA5E3PF7_hSnTND_eSRyK_A',
domain : u'ROOT', secretkey : u'gLz6lDVFch6ayBAB-8SMZ-RPiOgJTEVJmNgnAMdv36O6eMUKUN6R-Q-3ZtYJUH84jye-5q6Qpxf5WOtSuG8WtA',
iscallerchilddomain : False, state : u'enabled', accounttype : 1, id : u'a8b21230-bfac-11e5-be38-06a910012727',
isdefault : True, accountid : u'a8b2063c-bfac-11e5-be38-06a910012727'}]
2016-01-30 07:36:30,521 - DEBUG - ==== Test Client Creation Successful ====
-----------------------------------

Nothing actually happens ?!
Same with test_acl_listvm.py

Example(false positive-failure is in marvin test code):
-------------------------------------------------------------
nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM -a tags=advanced,required_hardware=true
/tmp/integration/component/test_browse_templates.py
2016-01-30 19:52:26,160 - CRITICAL - EXCEPTION: None: ['Traceback (most recent call last):\n',
' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run\n self.setUp()\n',
' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp\n self.setupContext(ancestor)\n',
' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext\n try_run(context,
names)\n', ' File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n
return func()\n', ' File "/tmp/integration/component/test_browse_templates.py", line 84, in
setUpClass\n cls.uploadurl=cls.testdata["configurableData"]["browser_upload_template"][cls.uploadtemplateformat]["url"]\n',
"KeyError: 'browser_upload_template'\n"]





<http://www.shapeblue.com/>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
    |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com<mailto:paul.angus@shapeblue.com> | t: @cloudyangus<mailto:paul.angus@shapeblue.com%20|%20t:%20@cloudyangus>
     |      w:      www.shapeblue.com<http://www.shapeblue.com/><http://www.shapeblue.com/>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK





Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India
LLP is a company incorporated in India and is operated under license from Shape Blue Ltd.
Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under
license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered
trademark.
This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error.




-----Original Message-----
From: Will Stevens [mailto:williamstevens@gmail.com]
Sent: Monday, March 14, 2016 4:26 AM
To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org><mailto:dev@cloudstack.apache.org>
Subject: Re: [PROPOSAL] Minimum Viable CI Integration

The latest version (0.2.0) of upr is now available...

https://github.com/cloudops/upr

All the details are in the README file of the repo, but the quick overview of the changes
include:

- Added the ability to comment on a pull request (by PR number or
commit).
- Implemented both S3 and Swift object storage backends for uploading
files (tested with AWS S3, Cloudian (S3) and Swift).
- Allow for automatic expiration of uploaded files to clean up the
object store after a period of time (in both S3 and Swift).
- Added the ability to override the default comment template with a
local file (look at the current template for details).

Let me know if you guys have questions, comments or feedback on that...

Hopefully the apache infra guys will get back to me tomorrow so we can better assess how we
need to move forward.

Cheers,

Will

On Sun, Mar 13, 2016 at 7:11 PM, Will Stevens <williamstevens@gmail.com<mailto:williamstevens@gmail.com><mailto:williamstevens@gmail.com>>
wrote:

Unfortunately this would only give us the ability to 'red
light'/'green light' pull requests via a distributed CI environments.
The reason I limited the request to this permission is because
anything they give us will be global to the entire apache org and this
permission is restrictive enough that we can't do damage elsewhere.
There is a bit of history of them denying all requests of this manner,
so I wanted to focus on our minimum viable use case, while limiting their liability, to improve
our chances.

You can follow the action here:
https://issues.apache.org/jira/browse/INFRA-11429

They have officially said no at this point, but I have asked them to
reconsider and asked them to provide suggestions for solving our problem.

If they say no again, they will be doing so with the full knowledge
that the decision leaves us with our hands tied and we have exhausted
our official options through them.

If we end up in this situation, we will have to get creative and I
suspect that will result in some activities moving outside the Apache
repo. I have not gone there (yet) since I feel that will need to be
voted on in an official thread of its own before we make a decision like that.

Feedback and suggestions are very welcome.

I am finalizing the 'comment' functionality in my tool 'upr' this
weekend, so that could give us some limited options. I will try to
push a release later tonight. It is all written and functional now,
just need to clean up the documentation and examples.

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> |
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software
Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
| CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>




DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite,
a Persistent Systems business. It is intended only for the use of the individual or entity
to which it is addressed. If you are not the intended recipient, you are not authorized to
read, retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent
Systems business does not accept any liability for virus infected mails.


DISCLAIMER ========== This e-mail may contain privileged and confidential information which
is the property of Accelerite, a Persistent Systems business. It is intended only for the
use of the individual or entity to which it is addressed. If you are not the intended recipient,
you are not authorized to read, retain, copy, print, distribute or use this message. If you
have received this communication in error, please notify the sender and delete all copies
of this message. Accelerite, a Persistent Systems business does not accept any liability for
virus infected mails.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message