continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Venisse" <emmanuel.veni...@gmail.com>
Subject Re: Distributed builds
Date Wed, 22 Oct 2008 12:37:41 GMT
Before to start on distributed builds (I want it in the next version), I
think we must work on some things before.
- the build system must be totally independant of the DB
- the build system must sent events for each steps so Continuum will know
the status and if the DB should be updated
- We must refactor the build system with a plugin mechanism like maven
(Olivier has started to work on a proposal)

Then we'll can play with distributed builds.
I added some comment below.

On Wed, Oct 22, 2008 at 4:07 AM, Marica Tan <ctan@exist.com> wrote:

> Hi,
>
> Our team in exist came up with a requirements for distributed builds.
>
> *A. Distribution of work*
>
>     Master and Slave Agent will communicate with each other through
> xml-rpc.

I'm not sure xmlrpc is the best choice for the communication.
1- the communication between the master and slaves must be secured
2- when Continuum master is updated (new version), I think it can be good to
update all slaves too

A solution would can be to store the agent binaries in the master, so if you
update the master, agent binaries will be updated too. For the
communication, the master would can open a ssh connection to the slave
machine and start the agent remotely, the agent can be installed by the
master automatically.
With that, the communication is totally secured, configured in the master
and the agent is always up-to-date with the master version.


>
>   1. Master Agent - controls build via xml-rpc
>      1. slaves need to be added to the master server
>         - attach the slave to a build environment as a type of
>         "installation". Each slave reports a number of available
> installations.
>      2. check availability of slaves
>      - slave is available if it's up and has no current build.
>         - if slave is available then dispatch build to the slave.
>         - if slave is unavailable then enqueue build on the master

I'm not totally agree. With a build/slaves mechanism, you can choose to
delegate the build to an available slave, but if it's the first time a build
is assigned for a specific project to this node, a full checkout will be
required on the node even if the checkout was already done on an other node.
The disk usage is potentially the size of all projects X nb nodes.
Some users will want to use always the same node for all builds of a project
or attach a specific node to a project group.

What about parallel build?

>
>         3. distribute work
>         - select slave based on build environment (master)
>         - build
>            - pass current configuration
>            - expect result (master)/return result (slave)
>            - attach each result returned by the slave to each project in
>            the group (add to database)
>          2. Slave Agent - is a dumb build agent that will run on another
>   build server. It will only build the project and return the result. It
> also
>   has it's own installation.
>      - queried for available installations
>      - start build, stop build
>      - returns build result
>

I think we'll need an "internal" remote repository used by all nodes. If the
build definition run only "mvn clean install", the artifact will be stored
on the local repository of the node and other nodes will can't use the new
artifact if we don't use our remote repository.

>
> *B. UI/Configuration*
>
>   1. Aggregated view of distributed projects


Can you add more informations? Will it modify the aspect of the actual
pages?

>
>   2. Slave server list (new page)
>   3. Build Environment
>      - add slave selection
>      - modify installations based on slave selection
>   4. distributed build configuration stored and editable in XML
>   configuration file
>
> Limitations
>
>   - credentials (s.a. svn credentials) are passed along if specified, but
>   if server cache is used it will need to be done individually on the
> slaves
>   - slaves can only be configured at the project group level, by a system
>   administrator
>   - project dependencies are not considered in distribution -
>   interdependent projects should target the same slave server
>
> Future Enhancements
>
>   1. Policy-based distribution
>      - next available
>      - load balanced
>      - target environment matching
>
>
> Comments, suggestions and/or violent reactions are welcome :)


Can you add the proposal in the wiki?
Thanks

Emmanuel

>
>
>
> Thanks,
> --
> Marica
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message