continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marica Tan" <c...@exist.com>
Subject Re: Distributed builds
Date Thu, 23 Oct 2008 12:16:26 GMT
On Wed, Oct 22, 2008 at 8:37 PM, Emmanuel Venisse <
emmanuel.venisse@gmail.com> wrote:

> Before to start on distributed builds (I want it in the next version), I
> think we must work on some things before.
> - the build system must be totally independant of the DB
> - the build system must sent events for each steps so Continuum will know
> the status and if the DB should be updated
> - We must refactor the build system with a plugin mechanism like maven
> (Olivier has started to work on a proposal)


> Then we'll can play with distributed builds.
> I added some comment below.
>
> On Wed, Oct 22, 2008 at 4:07 AM, Marica Tan <ctan@exist.com> wrote:
>
> > Hi,
> >
> > Our team in exist came up with a requirements for distributed builds.
> >
> > *A. Distribution of work*
> >
> >     Master and Slave Agent will communicate with each other through
> > xml-rpc.
>
> I'm not sure xmlrpc is the best choice for the communication.
> 1- the communication between the master and slaves must be secured
> 2- when Continuum master is updated (new version), I think it can be good
> to
> update all slaves too
>
> A solution would can be to store the agent binaries in the master, so if
> you
> update the master, agent binaries will be updated too. For the
> communication, the master would can open a ssh connection to the slave
> machine and start the agent remotely, the agent can be installed by the
> master automatically.
> With that, the communication is totally secured, configured in the master
> and the agent is always up-to-date with the master version.
>

IMHO, there may be other alternative for a secured communication. For
instance, we can wrap xmlrpc with SSL to make it secured.
I'm not sure if it's a good idea to use ssh just for a secured communication
and it's not yet clear to me how to trigger the build using ssh.

>
> >
> >   1. Master Agent - controls build via xml-rpc
> >      1. slaves need to be added to the master server
> >         - attach the slave to a build environment as a type of
> >         "installation". Each slave reports a number of available
> > installations.
> >      2. check availability of slaves
> >      - slave is available if it's up and has no current build.
> >         - if slave is available then dispatch build to the slave.
> >         - if slave is unavailable then enqueue build on the master
>
> I'm not totally agree. With a build/slaves mechanism, you can choose to
> delegate the build to an available slave, but if it's the first time a
> build
> is assigned for a specific project to this node, a full checkout will be
> required on the node even if the checkout was already done on an other
> node.
> The disk usage is potentially the size of all projects X nb nodes.
> Some users will want to use always the same node for all builds of a
> project
> or attach a specific node to a project group.


> What about parallel build?


We have another proposal for the parallel build.

>
> >
> >         3. distribute work
> >         - select slave based on build environment (master)
> >         - build
> >            - pass current configuration
> >            - expect result (master)/return result (slave)
> >            - attach each result returned by the slave to each project in
> >            the group (add to database)
> >          2. Slave Agent - is a dumb build agent that will run on another
> >   build server. It will only build the project and return the result. It
> > also
> >   has it's own installation.
> >      - queried for available installations
> >      - start build, stop build
> >      - returns build result
> >
>
> I think we'll need an "internal" remote repository used by all nodes. If
> the
> build definition run only "mvn clean install", the artifact will be stored
> on the local repository of the node and other nodes will can't use the new
> artifact if we don't use our remote repository.


Yes, I agree.

> >
> > *B. UI/Configuration*
> >
> >   1. Aggregated view of distributed projects
>
>
> Can you add more informations? Will it modify the aspect of the actual
> pages?


No. This is a new page listing all projects currently building and where
they are being built.

>
> >
> >   2. Slave server list (new page)
> >   3. Build Environment
> >      - add slave selection
> >      - modify installations based on slave selection
> >   4. distributed build configuration stored and editable in XML
> >   configuration file
> >
> > Limitations
> >
> >   - credentials (s.a. svn credentials) are passed along if specified, but
> >   if server cache is used it will need to be done individually on the
> > slaves
> >   - slaves can only be configured at the project group level, by a system
> >   administrator
> >   - project dependencies are not considered in distribution -
> >   interdependent projects should target the same slave server
> >
> > Future Enhancements
> >
> >   1. Policy-based distribution
> >      - next available
> >      - load balanced
> >      - target environment matching
> >
> >
> > Comments, suggestions and/or violent reactions are welcome :)
>
>
> Can you add the proposal in the wiki?
> Thanks
>
> Emmanuel
>
> >
> >
> >
> > Thanks,
> > --
> > Marica
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message