cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <wstev...@cloudops.com>
Subject Re: [PROPOSAL] Minimum Viable CI Integration
Date Fri, 04 Mar 2016 23:24:06 GMT
Alright, I have done some testing and here are the results.

I have improved the 'add comment' to pull request functionality to better
layout logs and such [1], but we may not use this functionality going
forward, but I will leave it in the tool regardless in case it is useful
later.

I have been able to programmatically update a pull request with status
results.  Here are a couple examples, [2], [3], notice that the original
failure is overwritten by the success when the same test is run again and
succeeds.

Even more interesting is the ability for different configurations to submit
their own results, so we can better isolate issues if more than one CI
environment is run against a PR.  Here is an example: [4]

Now for the crux of the issue.  The permissions needed in order to perform
all these tasks.  We would need a new access token create [6] with the very
stripped down permissions of repo:status and public_repo as seen here [5].

I feel we should be able to request an access token with these permissions
from whoever has control of the apache github account so we can do this
integration and provide it to the people who have CI environments who need
to post their status.  Do you guys think this would be possible?  If not, I
need to figure out what I need to do to write an official app so we can
just 'allow' the app and have that done transparently.

Let me know what you guys think...

Cheers,

Will

[1]
https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/pr_testing/text_log_summary.png
[2]
https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/pr_testing/failed_ci.png
[3]
https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/pr_testing/success_ci.png
[4]
https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/pr_testing/combo_ci.png
[5]
https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/pr_testing/permissions.png
[6] https://github.com/settings/tokens

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Fri, Mar 4, 2016 at 2:46 PM, Remi Bergsma <RBergsma@schubergphilis.com>
wrote:

> Hi Will,
>
> Posting in the comments is the easiest thing to do (and already far better
> then e-mailing IMHO), but we’ve seeing the past that these comments are
> easily overlooked and ignored. It’s best to have them integrated and
> registered as an integration test. That we never got to work (and it wasn’t
> a technical problem). So, the same way Jenkins and Travis are registered.
> It will show green or red and people merging are alerted in the scripts
> when they are about to merge something unstable (i.e. red). It’s also very
> hard to do any automation on it without.
>
> If that’s impossible due to permissions, you could go one level down and
> have Jenkins (the one that is in place already) call a set of slaves that
> runs all the different tests (hosted by community members and this can be
> totally different approaches). You’ll have an automated system, it will
> collect and archive both results and logs and stats of tests that fail
> regularly. Jenkins works great with nosetest results (so don’t reinvent
> that). You need this, because Marvin will fail every now and then and
> you’ll want to kick flaky tests out because otherwise people say “ah, that
> test sometimes fails” and ignore it. Risking to miss the real failure and
> rendering the whole thing useless. You won’t be able to see failure rates
> form comments.
>
> Let me give you some examples of Jenkins integration with Marvin:
>
> Example where you see all results in the console:
>
> https://beta-jenkins.mcc.schubergphilis.com/job/cosmic/job/run-integration-tests/6/console
>
>
> Example of test report with failures:
>
> https://beta-jenkins.mcc.schubergphilis.com/job/cosmic/job/tracking-repo-build/2/testReport/
>
> Full details per test:
>
> https://beta-jenkins.mcc.schubergphilis.com/job/cosmic/job/tracking-repo-build/2/testReport/integration.smoke.test_vm_life_cycle/TestVMLifeCycle/
>
>
> https://beta-jenkins.mcc.schubergphilis.com/job/cosmic/job/tracking-repo-build/2/testReport/integration.smoke.test_vm_life_cycle/TestVMLifeCycle/test_08_migrate_vm/
>
> This is work-in-progress. I hope you at least get the idea.
>
> Anyway, fine to get started like you described but it will be a lot of
> manual work. From experience I know that it won’t be sustainable.
>
> Regards,
> Remi
>
>
>
>
>
> On 04/03/16 05:42, "Will Stevens" <williamstevens@gmail.com> wrote:
>
> >Awesome, thanks for the update Bharat, that is great progress.
> >
> >If you need a hand with the posting back to the pull requests on github,
> >just let me know, I have that piece working.  My implementation is a cross
> >platform binary without any dependencies so it could be easy to integrate
> >if that would be helpful.  I plan to clean it up a bit and open source it.
> >I have only spent a few hours on it so far, so I want to do a once through
> >to get it ready to release.
> >
> >Just an FYI, it is not very easy to post back to github using the 'Pull
> >Requests' API because of the code reference requirements.  Checkout the
> >'Issues' API which is basically the same thing, but enables you to create
> >comments without having to specify a specific location in the PRs code
> >which you are commenting on.  This tripped me up for a while today and
> once
> >I realized I could do it through the Issues instead of the Pull Requests,
> >it made things infinitely easier.
> >
> >Let me know if you have other questions.
> >
> >Cheers,
> >
> >Will
> >
> >
> >On Thu, Mar 3, 2016 at 11:29 PM, Bharat Kumar <
> bharat.kumar@accelerite.com>
> >wrote:
> >
> >> Hi will,
> >>
> >> we have a solution to post the results to the community by email. We
> also
> >> have a github integration to fetch the Prs , run tests against them and
> post
> >> the consolidated results by email and share the logs using dropbox.
> >>
> >> We are facing some setup delays to get this up and running. I am sure we
> >> will be able to do this shortly.
> >>
> >> I am also working on posting the results in PR as comments.
> >>
> >> Thanks,
> >> Bharat.
> >>
> >>
> >>
> >> > On 04-Mar-2016, at 7:36 AM, Will Stevens <williamstevens@gmail.com>
> >> wrote:
> >> >
> >> > Last I knew Bharat did not have a solution for posting results back to
> >> the
> >> > community. I could be wrong though, I don't really know how complete a
> >> > solution Bharat has at this point.
> >> >
> >> > There are two other CI implementations in various states of
> completeness
> >> > and I think it is important to have a common mechanism which will
> work to
> >> > post results back to the community regardless of the CI
> implementation.
> >> >
> >> > "bubble" from SBP seems to be the most complete so far and has been
> used
> >> to
> >> > test over 100 PRs, but I don't think the results have formally
> >> contributed
> >> > back for the community to review.
> >> >
> >> > "trillian" is a project shapeblue is working on, but I am still
> getting
> >> > details for when it is expected to be operational.
> >> >
> >> > This may not be a final solution, but I feel it is a good first step
> >> which
> >> > will be easy for everyone to work with.
> >> > On Mar 3, 2016 8:35 PM, "Gandikota Srinivas" <
> >> gandikota.srinivas@gmail.com>
> >> > wrote:
> >> >
> >> >> Will,
> >> >> I guess Bharat has something similar in working.
> >> >>
> >> >> Bharat,
> >> >> Can you please elaborate your approach for sharing the CI results
> with
> >> >> community ?
> >> >>
> >> >> Thanks,
> >> >> Srinivas
> >> >>
> >> >> On Fri, Mar 4, 2016 at 6:28 AM, Will Stevens <
> williamstevens@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Apparently attached files don't work when sending to the mailing
> list.
> >> >>>
> >> >>> Find the screenshot here:
> >> >>>
> >> >>>
> >> >>
> >>
> https://objects-east.cloud.ca/v1/5ef827605f884961b94881e928e7a250/swill/Screen%20Shot%202016-03-03%20at%207.53.42%20PM.png
> >> >>>
> >> >>> On Thu, Mar 3, 2016 at 7:54 PM, Will Stevens <
> williamstevens@gmail.com
> >> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hey Everyone,
> >> >>>> As I am sure most of you are aware, I have been focusing a
lot on
> ways
> >> >> to
> >> >>>> get CI integrated back into the community.
> >> >>>>
> >> >>>> Today I build a little POC to validate some ideas and get a
feel
> for a
> >> >>>> potential approach for getting CI integrated into the Github
pull
> >> >> request
> >> >>>> workflow.
> >> >>>>
> >> >>>> There are multiple individuals/companies focusing on CI right
now
> >> >> (which
> >> >>>> is a good thing), but there has not really been much discussion
> (that
> >> I
> >> >>> am
> >> >>>> aware of) for how these different CI runs will push back results
to
> >> the
> >> >>>> community.  I want to make sure that nobody's work on this
topic
> goes
> >> >> to
> >> >>>> waste, so my goal is to provide a simple and consistent way
for
> >> >> everyone
> >> >>> to
> >> >>>> push their results back to the community.
> >> >>>>
> >> >>>> Here is the basic idea (please give feedback):
> >> >>>> - A simple cross platform command line tool with zero dependencies
> >> will
> >> >>> be
> >> >>>> provided (and open sourced) which will handle the submission
of
> the CI
> >> >>>> results back to the community.  It is written in Golang and
is
> >> >> currently
> >> >>>> called 'notify_pr'.
> >> >>>> - At the end of a CI execution, the CI should automate the
> execution
> >> of
> >> >>>> this tool to handle updating the appropriate PR with the results.
> >> >>>>
> >> >>>> Configuration can be done via the command line or through an
INI
> file.
> >> >>>> Here is an example of the configuration details.  The commit
is the
> >> >>>> commit that the CI just executed against.
> >> >>>>
> >> >>>> token  = <your github token>
> >> >>>> owner  = apache
> >> >>>> repo   = cloudstack
> >> >>>> commit = c8443496d3fad78a4b1a48a0992ce545bde299e8
> >> >>>>
> >> >>>> summary_file = <a text file summary of the run>
> >> >>>> full_detail_dir = <a directory structure to be recursively
> uploaded to
> >> >>>> object store>
> >> >>>> full_detail_files = <a comma separated list of files to
upload to
> >> >> object
> >> >>>> store>
> >> >>>> store_api = <swift or s3>
> >> >>>> store_endpoint = <url endpoint>
> >> >>>> store_identity = <keystone identity or aws access key>
> >> >>>> store_secret = <keystone password or aws secret key>
> >> >>>>
> >> >>>> I have not yet implemented the object storage endpoints, but
I have
> >> >> code
> >> >>>> to do it from a different project, so I just need to add it.
 I
> will
> >> be
> >> >>>> able to host my CI output in a swift object store, but others
may
> need
> >> >> to
> >> >>>> use AWS S3.  Maybe we can get sponsorship for this storage.
 We
> will
> >> >> only
> >> >>>> keep the logs for a window of like a week or so on the object
> store so
> >> >>> the
> >> >>>> storage usage will not be ever growing.
> >> >>>>
> >> >>>> Basically, the tool takes the details of the repository you
are
> >> >>> validating
> >> >>>> a Pull Request for and the commit you are building.  It also
takes
> the
> >> >>>> files you would like to push to the pull request.  The summary
file
> >> >> will
> >> >>> be
> >> >>>> shown as text in the pull request comment and the other files
will
> be
> >> >>>> uploaded to an object store and will be publically available
for a
> >> >> period
> >> >>>> of time (probably about a week and then get cleaned up, details
> TBD).
> >> >>> The
> >> >>>> files to be uploaded to object store could be either specified
as
> >> >>>> individual files, OR a target directory and all the files in
that
> >> >>> directory
> >> >>>> will be recursively uploaded to the object store.
> >> >>>>
> >> >>>> When the tool is run, it will scan through all the open pull
> requests
> >> >> in
> >> >>>> the target repository and when it finds the pull request
> corresponding
> >> >> to
> >> >>>> the commit in question, it will post the details as a comment
to
> that
> >> >>> pull
> >> >>>> request.  This functionality is currently working (see the
attached
> >> >>>> screenshot).  I can change the formatting and such, this is
just an
> >> >>> example.
> >> >>>>
> >> >>>> This is still a very rough concept that I have only worked
on for a
> >> >> day,
> >> >>>> but hopefully you guys agree that it is a good start towards
a
> >> >> consistent
> >> >>>> feedback mechanism for the community to take advantage of the
> >> different
> >> >>>> distributed CI installations.
> >> >>>>
> >> >>>> Please voice your feedback and concerns.  I am sure I have
not
> thought
> >> >> of
> >> >>>> everything and we may still want to make fundamental changes
to the
> >> >>>> approach, but I think the concept is solid.
> >> >>>>
> >> >>>> Cheers,
> >> >>>>
> >> >>>> Will
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
> >>
> >>
> >> DISCLAIMER
> >> ==========
> >> This e-mail may contain privileged and confidential information which is
> >> the property of Accelerite, a Persistent Systems business. It is
> intended
> >> only for the use of the individual or entity to which it is addressed.
> If
> >> you are not the intended recipient, you are not authorized to read,
> retain,
> >> copy, print, distribute or use this message. If you have received this
> >> communication in error, please notify the sender and delete all copies
> of
> >> this message. Accelerite, a Persistent Systems business does not accept
> any
> >> liability for virus infected mails.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message