continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olivier Lamy" <ol...@apache.org>
Subject Re: Distributed builds
Date Wed, 22 Oct 2008 13:05:14 GMT
2008/10/22 Emmanuel Venisse <emmanuel.venisse@gmail.com>:
> Before to start on distributed builds (I want it in the next version), I
> think we must work on some things before.
> - the build system must be totally independant of the DB
> - the build system must sent events for each steps so Continuum will know
> the status and if the DB should be updated
> - We must refactor the build system with a plugin mechanism like maven
> (Olivier has started to work on a proposal)
My bad only on a paper :-)
>
> Then we'll can play with distributed builds.
> I added some comment below.
>
> On Wed, Oct 22, 2008 at 4:07 AM, Marica Tan <ctan@exist.com> wrote:
>
>> Hi,
>>
>> Our team in exist came up with a requirements for distributed builds.
>>
>> *A. Distribution of work*
>>
>>     Master and Slave Agent will communicate with each other through
>> xml-rpc.
>
> I'm not sure xmlrpc is the best choice for the communication.
> 1- the communication between the master and slaves must be secured
> 2- when Continuum master is updated (new version), I think it can be good to
> update all slaves too
>
> A solution would can be to store the agent binaries in the master, so if you
> update the master, agent binaries will be updated too. For the
> communication, the master would can open a ssh connection to the slave
> machine and start the agent remotely, the agent can be installed by the
> master automatically.
> With that, the communication is totally secured, configured in the master
> and the agent is always up-to-date with the master version.
>
>
>>
>>   1. Master Agent - controls build via xml-rpc
>>      1. slaves need to be added to the master server
>>         - attach the slave to a build environment as a type of
>>         "installation". Each slave reports a number of available
>> installations.
>>      2. check availability of slaves
>>      - slave is available if it's up and has no current build.
>>         - if slave is available then dispatch build to the slave.
>>         - if slave is unavailable then enqueue build on the master
>
> I'm not totally agree. With a build/slaves mechanism, you can choose to
> delegate the build to an available slave, but if it's the first time a build
> is assigned for a specific project to this node, a full checkout will be
> required on the node even if the checkout was already done on an other node.
> The disk usage is potentially the size of all projects X nb nodes.
> Some users will want to use always the same node for all builds of a project
> or attach a specific node to a project group.
>
> What about parallel build?
>
>>
>>         3. distribute work
>>         - select slave based on build environment (master)
>>         - build
>>            - pass current configuration
>>            - expect result (master)/return result (slave)
>>            - attach each result returned by the slave to each project in
>>            the group (add to database)
>>          2. Slave Agent - is a dumb build agent that will run on another
>>   build server. It will only build the project and return the result. It
>> also
>>   has it's own installation.
>>      - queried for available installations
>>      - start build, stop build
>>      - returns build result
>>
>
> I think we'll need an "internal" remote repository used by all nodes. If the
> build definition run only "mvn clean install", the artifact will be stored
> on the local repository of the node and other nodes will can't use the new
> artifact if we don't use our remote repository.
>
>>
>> *B. UI/Configuration*
>>
>>   1. Aggregated view of distributed projects
>
>
> Can you add more informations? Will it modify the aspect of the actual
> pages?
>
>>
>>   2. Slave server list (new page)
>>   3. Build Environment
>>      - add slave selection
>>      - modify installations based on slave selection
>>   4. distributed build configuration stored and editable in XML
>>   configuration file
>>
>> Limitations
>>
>>   - credentials (s.a. svn credentials) are passed along if specified, but
>>   if server cache is used it will need to be done individually on the
>> slaves
>>   - slaves can only be configured at the project group level, by a system
>>   administrator
>>   - project dependencies are not considered in distribution -
>>   interdependent projects should target the same slave server
>>
>> Future Enhancements
>>
>>   1. Policy-based distribution
>>      - next available
>>      - load balanced
>>      - target environment matching
>>
>>
>> Comments, suggestions and/or violent reactions are welcome :)
>
>
> Can you add the proposal in the wiki?
> Thanks
>
> Emmanuel
>
>>
>>
>>
>> Thanks,
>> --
>> Marica
>>
>

Mime
View raw message