continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Venisse" <emmanuel.veni...@gmail.com>
Subject Re: Distributed builds
Date Thu, 23 Oct 2008 13:38:46 GMT
On Thu, Oct 23, 2008 at 2:16 PM, Marica Tan <ctan@exist.com> wrote:

> On Wed, Oct 22, 2008 at 8:37 PM, Emmanuel Venisse <
> emmanuel.venisse@gmail.com> wrote:
>
> > Before to start on distributed builds (I want it in the next version), I
> > think we must work on some things before.
> > - the build system must be totally independant of the DB
> > - the build system must sent events for each steps so Continuum will know
> > the status and if the DB should be updated
> > - We must refactor the build system with a plugin mechanism like maven
> > (Olivier has started to work on a proposal)
>
>
> > Then we'll can play with distributed builds.
> > I added some comment below.
> >
> > On Wed, Oct 22, 2008 at 4:07 AM, Marica Tan <ctan@exist.com> wrote:
> >
> > > Hi,
> > >
> > > Our team in exist came up with a requirements for distributed builds.
> > >
> > > *A. Distribution of work*
> > >
> > >     Master and Slave Agent will communicate with each other through
> > > xml-rpc.
> >
> > I'm not sure xmlrpc is the best choice for the communication.
> > 1- the communication between the master and slaves must be secured
> > 2- when Continuum master is updated (new version), I think it can be good
> > to
> > update all slaves too
> >
> > A solution would can be to store the agent binaries in the master, so if
> > you
> > update the master, agent binaries will be updated too. For the
> > communication, the master would can open a ssh connection to the slave
> > machine and start the agent remotely, the agent can be installed by the
> > master automatically.
> > With that, the communication is totally secured, configured in the master
> > and the agent is always up-to-date with the master version.
> >
>
> IMHO, there may be other alternative for a secured communication. For
> instance, we can wrap xmlrpc with SSL to make it secured.
> I'm not sure if it's a good idea to use ssh just for a secured
> communication
> and it's not yet clear to me how to trigger the build using ssh.


The process is:

if latest agent version isn't deployed on the agent server
    copy the agent to the agent server
end if
start ssh session
launch agent with the project context (agent is a simple standalone to start
when needed, it isn't a service)
stop ssh session

With this mechanism, we get in real time the output of the agent instead of
to get the build result at the end. It's important to can to follow the
build output during the build so the build result page is updated
automatically.


>
> >
> > >
> > >   1. Master Agent - controls build via xml-rpc
> > >      1. slaves need to be added to the master server
> > >         - attach the slave to a build environment as a type of
> > >         "installation". Each slave reports a number of available
> > > installations.
> > >      2. check availability of slaves
> > >      - slave is available if it's up and has no current build.
> > >         - if slave is available then dispatch build to the slave.
> > >         - if slave is unavailable then enqueue build on the master
> >
> > I'm not totally agree. With a build/slaves mechanism, you can choose to
> > delegate the build to an available slave, but if it's the first time a
> > build
> > is assigned for a specific project to this node, a full checkout will be
> > required on the node even if the checkout was already done on an other
> > node.
> > The disk usage is potentially the size of all projects X nb nodes.
> > Some users will want to use always the same node for all builds of a
> > project
> > or attach a specific node to a project group.
>
>
> > What about parallel build?
>
>
> We have another proposal for the parallel build.
>
> >
> > >
> > >         3. distribute work
> > >         - select slave based on build environment (master)
> > >         - build
> > >            - pass current configuration
> > >            - expect result (master)/return result (slave)
> > >            - attach each result returned by the slave to each project
> in
> > >            the group (add to database)
> > >          2. Slave Agent - is a dumb build agent that will run on
> another
> > >   build server. It will only build the project and return the result.
> It
> > > also
> > >   has it's own installation.
> > >      - queried for available installations
> > >      - start build, stop build
> > >      - returns build result
> > >
> >
> > I think we'll need an "internal" remote repository used by all nodes. If
> > the
> > build definition run only "mvn clean install", the artifact will be
> stored
> > on the local repository of the node and other nodes will can't use the
> new
> > artifact if we don't use our remote repository.
>
>
> Yes, I agree.
>
> > >
> > > *B. UI/Configuration*
> > >
> > >   1. Aggregated view of distributed projects
> >
> >
> > Can you add more informations? Will it modify the aspect of the actual
> > pages?
>
>
> No. This is a new page listing all projects currently building and where
> they are being built.
>
> >
> > >
> > >   2. Slave server list (new page)
> > >   3. Build Environment
> > >      - add slave selection
> > >      - modify installations based on slave selection
> > >   4. distributed build configuration stored and editable in XML
> > >   configuration file
> > >
> > > Limitations
> > >
> > >   - credentials (s.a. svn credentials) are passed along if specified,
> but
> > >   if server cache is used it will need to be done individually on the
> > > slaves
> > >   - slaves can only be configured at the project group level, by a
> system
> > >   administrator
> > >   - project dependencies are not considered in distribution -
> > >   interdependent projects should target the same slave server
> > >
> > > Future Enhancements
> > >
> > >   1. Policy-based distribution
> > >      - next available
> > >      - load balanced
> > >      - target environment matching
> > >
> > >
> > > Comments, suggestions and/or violent reactions are welcome :)
> >
> >
> > Can you add the proposal in the wiki?
> > Thanks
> >
> > Emmanuel
> >
> > >
> > >
> > >
> > > Thanks,
> > > --
> > > Marica
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message