cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: [PROPOSAL] Minimum Viable CI Integration
Date Wed, 09 Mar 2016 16:41:38 GMT
Not really Will, Though I like the key per user idea. (positive feedback
doesn't count does it?)

On Wed, Mar 9, 2016 at 5:21 PM, Will Stevens <wstevens@cloudops.com> wrote:

> Anyone have any feedback on this?  I would like to get this ticket opened
> this week.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, Mar 8, 2016 at 12:46 PM, Will Stevens <wstevens@cloudops.com>
> wrote:
>
> > I am going to open a ticket with the infra team to request access tokens
> > for each of the organizations who are putting up hardware for the CI
> cause.
> >
> > The reason I am planning to request a token for each organization
> > individually is because I want to make a point about our need for CI and
> > the need for many groups to be doing CI in order for us to get full
> > hardware coverage of the different features we support.  The token is
> > needed in order for the different CI implementations to be able to simply
> > post back a 'red light', 'green light' status of the pull request in
> > github.  More details can be added in comments using an implementation
> like
> > what Bharat is working on, but the focus of this request will be to
> enable
> > us to have the ability for the different CI environments to get at least
> > 'red light', 'green light' functionality for pull requests.
> >
> > I will be requesting a token with the following basic permissions:
> > - public_repo - the same as an anonymous user
> > - repo:status - this allows for the status of a PR to be changed to one
> > of the following 4 states; pending, success, failure, error, with some
> > additional information such as; context, description, and a public url
> for
> > more details of the status.
> >
> > I am going to associate people with each token request and ask that each
> > person be sent their own token since they are providing infrastructure to
> > operate a CI environment and the community would like visibility into
> their
> > results.
> >
> > The people I am going to ask for a token for are as follows:
> >
> > Will Stevens <wstevens@cloudops.com>
> > Paul Angus <paul.angus@shapeblue.com>
> > Bharat Kumar <bharat.kumar@accelerite.com>
> > Remi Bergsma <RBergsma@schubergphilis.com>
> >
> > If you would also like to be included in this request and get your own
> > token in order to be able to contribute back the status of your CI runs,
> > please let me know so I can request a token for you as well.  Ilya, you
> had
> > shown some interest in this as well, would you like me to include you?
> If
> > so which email address should I use?
> >
> > Anyone else who is planning to contribute an environment to the CI
> process
> > in the near to medium future?
> >
> > Cheers,
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Mar 7, 2016 at 6:36 PM, Erik Weber <terbolous@gmail.com> wrote:
> >
> >> I guess the appropriate channel would be to create a jira ticket for
> >> INFRA:
> >> https://issues.apache.org/jira/browse/INFRA
> >>
> >> --
> >> Erik
> >>
> >> On Mon, Mar 7, 2016 at 11:18 PM, Will Stevens <wstevens@cloudops.com>
> >> wrote:
> >>
> >> > This is kind of what I was expecting.  Do you know who I would be
> >> > contacting?  The permissions required are VERY minimal AND they have
> >> > already given the 'TravisCI' application the same permissions as we
> need
> >> > for this.
> >> >
> >> > How did we get the TravisCI application enabled and the permissions
> >> > accepted for that integration?
> >> >
> >> > I have been trying to thinking of ways to potentially work the system
> >> from
> >> > that side.  The main issue I have with this approach is that it is
> not a
> >> > single application that we want to give permission to.  Ideally, we
> >> would
> >> > give each individual/organization who is contributing a CI environment
> >> > their own token.  In that case, we would have to register the CI of
> each
> >> > party as their own CI integration.
> >> >
> >> > Here are some ideas I have as a 'work around' to the problem:
> >> >
> >> > 1) Register a single upr CI application with Github and have the
> apache
> >> > guys enable the integration.  This will give the application a single
> >> > access token.  I can then compile upr with the access token embedded
> >> into
> >> > the binary.  I don't like this approach and I feel we would probably
> be
> >> > violating some ToS somewhere.
> >> >
> >> > 2) Provide a web server implementation that each CI party can use to
> >> > register their own CI endpoint as a Github application integration.
> >> Then
> >> > we have the apache guys enable each of them (which will be a harder
> >> sell to
> >> > them), but then each CI party will get their own token and will be
> able
> >> to
> >> > post back as themselves.  This is also nice because if someone is not
> a
> >> > good community member and misbehaves, their integration can be revoked
> >> > without it affecting everyone else who has a CI integration.
> >> >
> >> > 3) Provide a single web server implementation that is registered as a
> >> > Github Application Integration.  This implementation is then approved
> by
> >> > the apache guys for the cloudstack repo.  This web server
> implementation
> >> > (let's call it upr_server) keeps our one and only access token.  I
> then
> >> > modify the upr command line tool to take a token that is provided by
> the
> >> > upr_server when a CI party registers on the upr_server website.  The
> >> > upr command
> >> > will actually target the upr_server box when posting statuses, etc,
> >> which
> >> > will essentially proxy the calls on to Github using the token that was
> >> > approved for the integration.
> >> >
> >> > I think that 3) is probably the cleanest solution and would reduce our
> >> > chances of getting banned by someone for being too cheeky.  It is a
> >> whole
> >> > lot of trouble for nothing, but if they are going to be a stick in the
> >> mud
> >> > about this and not allow access tokens to anything other than official
> >> > github supported integrations, then I will have to make that work...
> >> >
> >> > Ideas?  Thoughts?  Rants?  :P
> >> >
> >> > *Will STEVENS*
> >> > Lead Developer
> >> >
> >> > *CloudOps* *| *Cloud Solutions Experts
> >> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> > w cloudops.com *|* tw @CloudOps_
> >> >
> >> > On Mon, Mar 7, 2016 at 4:12 PM, Remi Bergsma <
> >> RBergsma@schubergphilis.com>
> >> > wrote:
> >> >
> >> > > Hi Will,
> >> > >
> >> > > This is the main problem: there’s no one except Apache Infra with
> >> access
> >> > > to the Github CloudStack repo. Even committers have to push to
> Apache
> >> > git,
> >> > > which is mirrored to Github. We can’t close a PR, set a label,
> change
> >> a
> >> > > title or whatever basic operation. You can ask them for a token.
> When
> >> I
> >> > (as
> >> > > the release manager) asked for any more permission than an anonymous
> >> user
> >> > > has it was kindly refused. I really hope you’ll have more luck but
I
> >> > > wouldn’t count on it.
> >> > >
> >> > >
> >> > >
> >> > > Regards,
> >> > > Remi
> >> > >
> >> > >
> >> > > On 07/03/16 19:10, "williamstevens@gmail.com on behalf of Will
> >> Stevens"
> >> > <
> >> > > williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> >> > >
> >> > > >The main thing we have to sort out with this type of integration
> (as
> >> it
> >> > is
> >> > > >today) is the distribution of access tokens with the correct
> >> permissions
> >> > > on
> >> > > >the apache/cloudstack github repo.  The required permissions are
> very
> >> > > >limited, but I don't know if we have access to create new tokens.
> >> If we
> >> > > >don't then I will have to develop an application integration
> >> workaround
> >> > to
> >> > > >make it easier for the people with access to the apache/cloudstack
> >> repo
> >> > to
> >> > > >give the people running CI integrations access to update statuses
> >> (like
> >> > > the
> >> > > >current travis integration).
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > >If you have questions or feedback, please don't be shy.
> >> > > >
> >> > > >*Will STEVENS*
> >> > > >Lead Developer
> >> > > >
> >> > > >*CloudOps* *| *Cloud Solutions Experts
> >> > > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> > > >w cloudops.com *|* tw @CloudOps_
> >> > > >
> >> > > >On Mon, Mar 7, 2016 at 12:36 PM, Bharat Kumar <
> >> > > bharat.kumar@accelerite.com>
> >> > > >wrote:
> >> > > >
> >> > > >> Hi guys,
> >> > > >>
> >> > > >> I am also working on the similar reporting problem, here
is what
> i
> >> did
> >> > > >>
> >> > > >> link to the report
> https://github.com/bvbharatk/cloud-stack/pull/1
> >> > > >>
> >> > > >> I am thinking this is good enough for now, I want to start
> posting
> >> the
> >> > > >> results on each pr as shown in the above link.
> >> > > >> please give me your comments or suggestions.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Bharat
> >> > > >> On 05-Mar-2016, at 7:02 PM, Will Stevens <wstevens@cloudops.com
> >> > <mailto:
> >> > > >> wstevens@cloudops.com>> wrote:
> >> > > >>
> >> > > >> Daan
> >> > > >>
> >> > > >> Regarding the obligatory provider id.  I agree, but I am
still
> >> trying
> >> > to
> >> > > >> figure out the details.  Creating distinct runs that have
their
> own
> >> > > status
> >> > > >> is done by setting the 'context'.  I think we would need
to have
> >> two
> >> > > pieces
> >> > > >> to this.  A provider id and an environment id.
> >> > > >>
> >> > > >> So for example.  Lets assume that my provider id is 'CloudOps'
> and
> >> I
> >> > > have
> >> > > >> two different environments, one for 'KVM' and one for 'Xen'
(for
> >> > > example).
> >> > > >> I would then the tool would produce the following two independent
> >> CI
> >> > > >> statuses.
> >> > > >> 'CloudOps - KVM' : with a basic description of the environment.
> >> > > >> 'CloudOps - Xen' : again with a basic description of the
env.
> >> > > >> ... and so on ...
> >> > > >>
> >> > > >> I am still sorting out the details here as well as making
it easy
> >> for
> >> > > us to
> >> > > >> integrate this into the apache/cloudstack repo with the access
we
> >> > > currently
> >> > > >> have.  Adding this is 'no biggy' for me because I am building
> this
> >> > tool
> >> > > as
> >> > > >> we speak, and trying to tailor it to our (ACS) needs, so
this
> type
> >> of
> >> > > >> feedback is perfect as it allows me to adapt the tool before
I
> get
> >> too
> >> > > >> deep.
> >> > > >>
> >> > > >> *Will STEVENS*
> >> > > >> Lead Developer
> >> > > >>
> >> > > >> *CloudOps* *| *Cloud Solutions Experts
> >> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_
> >> > > >>
> >> > > >> On Sat, Mar 5, 2016 at 4:17 AM, Daan Hoogland <
> >> > daan.hoogland@gmail.com
> >> > > >> <mailto:daan.hoogland@gmail.com>>
> >> > > >> wrote:
> >> > > >>
> >> > > >> Will
> >> > > >>
> >> > > >> Gret work, especially the thing you are showing in link [4],
I
> >> would
> >> > > like
> >> > > >> to make an enhancement request and that is a obligatory provider
> >> id.
> >> > > Only
> >> > > >> if it is no biggy for you!
> >> > > >> Several people may decide to do a XVM on ChildrensOS for
instance
> >> and
> >> > > so we
> >> > > >> may be aware of an obscurity that is different. If one fails
and
> >> the
> >> > > other
> >> > > >> succeeds it is easily identified.
> >> > > >>
> >> > > >> Ilya,
> >> > > >>
> >> > > >> I have been playing with go and it is a very nice language
for
> >> such a
> >> > > >> simple script, though it wasn't exactly designed for it.
So don't
> >> read
> >> > > my
> >> > > >> comment/question as an objection. But we do have
> >> > > >> bash,python,c#,java,javascript,xslt,sql at least. That is
not
> >> counting
> >> > > the
> >> > > >> build system and I am sure the hyperv has some extra windows
> >> specific
> >> > > >> stuff.
> >> > > >> To me it is inherent to the nature of across platform
> orchestration
> >> > and
> >> > > >> provisioning system so it is fine. It is something to consider.
> On
> >> the
> >> > > >> other hand when bringing in new tools we don't make the choice
> >> so....
> >> > I
> >> > > am
> >> > > >> ranting, I guess.
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On Sat, Mar 5, 2016 at 7:38 AM, ilya <
> ilya.mailing.lists@gmail.com
> >> > > <mailto:
> >> > > >> ilya.mailing.lists@gmail.com>> wrote:
> >> > > >>
> >> > > >> I see where Daan is coming from :)  I thought this would
be 4th,
> >> not
> >> > > >> exactly 7ths.
> >> > > >>
> >> > > >> I'm not against golang by any means (if anything - its my
next
> >> "go" to
> >> > > >> language these days).
> >> > > >>
> >> > > >> Things to consider:
> >> > > >>
> >> > > >> Would notify-pr support proxy? I've been thinking on ways
of
> >> > > >> contributing test runs, there would have to be few things
i'd
> need
> >> to
> >> > > do.
> >> > > >>
> >> > > >> 1) massage the log content - such that no environment details
are
> >> > > >> exposed, i can probably handle this with sed/awk..
> >> > > >>
> >> > > >> 2) i'm behind multiple firewalls with no internet access.
> however,
> >> > some
> >> > > >> lab environments might have a proxy, so it would be nice
to have
> a
> >> > > >> support for it.
> >> > > >>
> >> > > >> Thanks
> >> > > >> ilya
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> On 3/4/16 6:56 AM, Will Stevens wrote:
> >> > > >> Yes, I have most of it already built and will be releasing
it
> later
> >> > > >> today
> >> > > >> or over the weekend.  The reason I chose Golang is because
it can
> >> be
> >> > > >> cross
> >> > > >> compiled to be run on any system and distributed as a single
> binary
> >> > > >> with
> >> > > >> no
> >> > > >> dependencies.  This means that no one will have to worry
about
> >> > building
> >> > > >> it
> >> > > >> or having to change their environment at all in order to
use it.
> >> I am
> >> > > >> trying to lower the barrier to entry and make it as easy
as
> >> possible
> >> > > >> for
> >> > > >> people to contribute back their CI execution details.
> >> > > >>
> >> > > >> *Will STEVENS*
> >> > > >> Lead Developer
> >> > > >>
> >> > > >> *CloudOps* *| *Cloud Solutions Experts
> >> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_
> >> > > >>
> >> > > >> On Fri, Mar 4, 2016 at 8:53 AM, Daan Hoogland <
> >> > daan.hoogland@gmail.com
> >> > > >> <mailto:daan.hoogland@gmail.com>
> >> > > >>
> >> > > >> wrote:
> >> > > >>
> >> > > >> Will, Do you have an implementation of notify-pr? I am asking
as
> >> you
> >> > > >> specify it will be implemented in golang which seems odd.
It is
> not
> >> > > >> amongst
> >> > > >> the 7 or so languages already in use.
> >> > > >>
> >> > > >> On Fri, Mar 4, 2016 at 1:54 AM, Will Stevens <
> >> > > >> williamstevens@gmail.com<mailto:williamstevens@gmail.com>>
> >> > > >> wrote:
> >> > > >>
> >> > > >> Hey Everyone,
> >> > > >> As I am sure most of you are aware, I have been focusing
a lot on
> >> > > >> ways
> >> > > >> to
> >> > > >> get CI integrated back into the community.
> >> > > >>
> >> > > >> Today I build a little POC to validate some ideas and get
a feel
> >> for
> >> > > >> a
> >> > > >> potential approach for getting CI integrated into the Github
pull
> >> > > >> request
> >> > > >> workflow.
> >> > > >>
> >> > > >> There are multiple individuals/companies focusing on CI right
now
> >> > > >> (which
> >> > > >> is a good thing), but there has not really been much discussion
> >> > > >> (that I
> >> > > >> am
> >> > > >> aware of) for how these different CI runs will push back
results
> to
> >> > > >> the
> >> > > >> community.  I want to make sure that nobody's work on this
topic
> >> goes
> >> > > >> to
> >> > > >> waste, so my goal is to provide a simple and consistent way
for
> >> > > >> everyone
> >> > > >> to
> >> > > >> push their results back to the community.
> >> > > >>
> >> > > >> Here is the basic idea (please give feedback):
> >> > > >> - A simple cross platform command line tool with zero
> dependencies
> >> > > >> will
> >> > > >> be
> >> > > >> provided (and open sourced) which will handle the submission
of
> the
> >> > > >> CI
> >> > > >> results back to the community.  It is written in Golang and
is
> >> > > >> currently
> >> > > >> called 'notify_pr'.
> >> > > >> - At the end of a CI execution, the CI should automate the
> >> execution
> >> > > >> of
> >> > > >> this tool to handle updating the appropriate PR with the
results.
> >> > > >>
> >> > > >> Configuration can be done via the command line or through
an INI
> >> > > >> file.
> >> > > >> Here is an example of the configuration details.  The commit
is
> the
> >> > > >> commit that the CI just executed against.
> >> > > >>
> >> > > >> token  = <your github token>
> >> > > >> owner  = apache
> >> > > >> repo   = cloudstack
> >> > > >> commit = c8443496d3fad78a4b1a48a0992ce545bde299e8
> >> > > >>
> >> > > >> summary_file = <a text file summary of the run>
> >> > > >> full_detail_dir = <a directory structure to be recursively
> uploaded
> >> > > >> to
> >> > > >> object store>
> >> > > >> full_detail_files = <a comma separated list of files to
upload to
> >> > > >> object
> >> > > >> store>
> >> > > >> store_api = <swift or s3>
> >> > > >> store_endpoint = <url endpoint>
> >> > > >> store_identity = <keystone identity or aws access key>
> >> > > >> store_secret = <keystone password or aws secret key>
> >> > > >>
> >> > > >> I have not yet implemented the object storage endpoints,
but I
> have
> >> > > >> code
> >> > > >> to do it from a different project, so I just need to add
it.  I
> >> will
> >> > > >> be
> >> > > >> able to host my CI output in a swift object store, but others
may
> >> > > >> need
> >> > > >> to
> >> > > >> use AWS S3.  Maybe we can get sponsorship for this storage.
 We
> >> will
> >> > > >> only
> >> > > >> keep the logs for a window of like a week or so on the object
> store
> >> > > >> so
> >> > > >> the
> >> > > >> storage usage will not be ever growing.
> >> > > >>
> >> > > >> Basically, the tool takes the details of the repository you
are
> >> > > >> validating
> >> > > >> a Pull Request for and the commit you are building.  It also
> takes
> >> > > >> the
> >> > > >> files you would like to push to the pull request.  The summary
> file
> >> > > >> will
> >> > > >> be
> >> > > >> shown as text in the pull request comment and the other files
> will
> >> be
> >> > > >> uploaded to an object store and will be publically available
for
> a
> >> > > >> period
> >> > > >> of time (probably about a week and then get cleaned up, details
> >> TBD).
> >> > > >> The
> >> > > >> files to be uploaded to object store could be either specified
as
> >> > > >> individual files, OR a target directory and all the files
in that
> >> > > >> directory
> >> > > >> will be recursively uploaded to the object store.
> >> > > >>
> >> > > >> When the tool is run, it will scan through all the open pull
> >> requests
> >> > > >> in
> >> > > >> the target repository and when it finds the pull request
> >> > > >> corresponding
> >> > > >> to
> >> > > >> the commit in question, it will post the details as a comment
to
> >> that
> >> > > >> pull
> >> > > >> request.  This functionality is currently working (see the
> attached
> >> > > >> screenshot).  I can change the formatting and such, this
is just
> an
> >> > > >> example.
> >> > > >>
> >> > > >> This is still a very rough concept that I have only worked
on
> for a
> >> > > >> day,
> >> > > >> but hopefully you guys agree that it is a good start towards
a
> >> > > >> consistent
> >> > > >> feedback mechanism for the community to take advantage of
the
> >> > > >> different
> >> > > >> distributed CI installations.
> >> > > >>
> >> > > >> Please voice your feedback and concerns.  I am sure I have
not
> >> > > >> thought
> >> > > >> of
> >> > > >> everything and we may still want to make fundamental changes
to
> the
> >> > > >> approach, but I think the concept is solid.
> >> > > >>
> >> > > >> Cheers,
> >> > > >>
> >> > > >> Will
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Daan
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Daan
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> DISCLAIMER
> >> > > >> ==========
> >> > > >> This e-mail may contain privileged and confidential information
> >> which
> >> > is
> >> > > >> the property of Accelerite, a Persistent Systems business.
It is
> >> > > intended
> >> > > >> only for the use of the individual or entity to which it
is
> >> addressed.
> >> > > If
> >> > > >> you are not the intended recipient, you are not authorized
to
> read,
> >> > > retain,
> >> > > >> copy, print, distribute or use this message. If you have
received
> >> this
> >> > > >> communication in error, please notify the sender and delete
all
> >> copies
> >> > > of
> >> > > >> this message. Accelerite, a Persistent Systems business does
not
> >> > accept
> >> > > any
> >> > > >> liability for virus infected mails.
> >> > > >>
> >> > >
> >> >
> >>
> >
> >
>



-- 
Daan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message