continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesse McConnell" <jesse.mcconn...@gmail.com>
Subject Re: build scheduling issues
Date Thu, 09 Nov 2006 15:20:19 GMT
sorry, once changed how?  once the group dependencies are sorted out?

if we are taking all projects across all groups and trying to generate
a cycle free graph then cycles are potentially a problem.

just so we are clear in my mind:

* grab all projects across all groups
* attempt to sort
* if success, build normally
* if failure, notify of cycle and proceed to sequentially process
groups, sorting projects in that group only and building
* if cycle detected in that group then build all projects in that
group in any old order that comes out of the database

The difference between this and what used to exist before my group
changes is that it will attempt to order inside the group and build as
opposed to before where it would just failover to building all
projects in whatever order came from the db.

is that where we are with this?

jesse



On 11/8/06, Brett Porter <brett@apache.org> wrote:
> That's a cycle between the groups, not the projects, so shouldn't be
> a problem once changed.
>
> - Brett
>
> On 09/11/2006, at 1:57 AM, Jesse McConnell wrote:
>
> > the issue isn't cycles within a normal m2 build, but when you shove 4
> > or 5 of them together into one continuum instance and then ask them
> > all to play together.
> >
> > For example if you shoved continuum, maven-shared and archiva all into
> > the same continuum instance and triggered a build with all of them in
> > the same directed graph then we have had instances of cycles in the
> > past (not sure about right now, mpir cycle might be refactored out
> > thanks to joakim).
> >
> > emm had another example last night when we were talking about it as
> > well.
> >
> > jesse
> >
> > On 11/8/06, Brett Porter <brett@apache.org> wrote:
> >> Sorry, am I missing something? How do we end up with a cycle in
> >> Continuum? Is there a specific example?
> >>
> >> I know it's possible - we had to allow it in the repository (eg,
> >> dom4j <-> jaxen), but it is certainly undesirable and honestly should
> >> be rare, especially in m2 built artifacts. Should it produce a build
> >> warning?
> >>
> >> Basically, the treatment there is to just stop following the tree
> >> when you hit the cycle, rather than changing the way things are
> >> ordered. So it's really arbitrary which might come first, but isn't
> >> really a concern there.
> >>
> >> Anyway, just wondering.
> >>
> >> Cheers,
> >> Brett
> >>
> >> On 08/11/2006, at 11:46 PM, Emmanuel Venisse wrote:
> >>
> >> > Yes
> >> >
> >> > Jesse McConnell a écrit :
> >> >> we should add a page that analyzes each schedule for cycles...that
> >> >> would be a cool little feature
> >> >> On 11/8/06, Emmanuel Venisse <emmanuel@venisse.net> wrote:
> >> >>> Yes, we need a global ordering, so projects will be build
> >> >>> independently of groups, because in some
> >> >>> case a cycle can be created between groups (not necessary between
> >> >>> projects).
> >> >>>
> >> >>> In case a cycle is detected between projects, continuum can't
> >> >>> find the build order. In this case, I
> >> >>> think we need to sort a little project so will reduce build
> >> >>> errors. So if we have a cycle, we can
> >> >>> sort projects in a group and build them. In most of case (maven
> >> >>> projects), we don't have a cycle in
> >> >>> a group.
> >> >>>
> >> >>> Emmanuel
> >> >>>
> >> >>> Brett Porter a écrit :
> >> >>> > I think you want global ordering. Grouping should just be
a
> >> >>> > display/management technique, not anything that changes how
> >> >>> projects are
> >> >>> > handled.
> >> >>> >
> >> >>> > However, this needs to be reviewed as a whole (which I think
> >> >>> Emmanuel is
> >> >>> > doing), such that builds can be triggered when their
> >> >>> dependencies change
> >> >>> > which will help with the ordering as it won't be dependant
on
> >> >>> them all
> >> >>> > being triggered at the same time?
> >> >>> >
> >> >>> > - Brett
> >> >>> >
> >> >>> > On 08/11/2006, at 9:51 AM, Jesse McConnell wrote:
> >> >>> >
> >> >>> >> I was reading through the DefaultContinuum.buildProjects
> >> >>> ( Schedule id
> >> >>> >> ) method and after discussing some things with Emmanuel...I
> >> >>> think we
> >> >>> >> have a problem here.  When I went through and refactored
> >> >>> things to
> >> >>> >> support a more Project Group centric setup with continuum
I
> >> >>> changed
> >> >>> >> this method a bit.
> >> >>> >>
> >> >>> >> Originally, this method would gather up all projects that
> >> >>> would be
> >> >>> >> triggered by that schedule, run them all through the project
> >> >>> sorter
> >> >>> >> and then build each in sequence.
> >> >>> >>
> >> >>> >> When I added the project groups to this mix, I changed
things
> >> >>> to be on
> >> >>> >> a project group basis, so that on a project group by project
> >> >>> group
> >> >>> >> basis it would order the projects and build them.  At
the
> >> time I
> >> >>> >> thought this was the way to go...but maybe not.
> >> >>> >>
> >> >>> >> 17:14 <evenisse> we need to take all projects from
all groups,
> >> >>> sort them
> >> >>> >> 17:15 <evenisse> if we don't have a cycle, it's
ok and we
> >> >>> build all
> >> >>> >> 17:15 <evenisse> if it isn't ok, we sort project
by group
> >> >>> >>
> >> >>> >> For example, if we loaded up a Plexus group and a Maven
> >> >>> group...the
> >> >>> >> way it currently is (with my change) it would process
all
> >> >>> triggered
> >> >>> >> builds within one group and then process all triggered
builds
> >> >>> in the
> >> >>> >> other group.   This would not take into account potential
> >> >>> dependencies
> >> >>> >> between the two.
> >> >>> >>
> >> >>> >> Does anyone have any thoughts on this?  I am inclined
to fix
> >> >>> it up so
> >> >>> >> its like it used to be where all projects across all project
> >> >>> groups
> >> >>> >> are thrown into the graph....I keep feeling like I am
missing
> >> >>> >> something wrong with this, but I can't pin it down.
> >> >>> >>
> >> >>> >> One thing that perhaps Emmanuel could explain a bit more
is
> >> >>> the third
> >> >>> >> comment there.  In our conversation on this he said that
he
> >> >>> thinks
> >> >>> >> that the cycles are cropping up all the time, and if thats
the
> >> >>> case
> >> >>> >> then we are building a lot of unordered builds which would
> >> >>> account for
> >> >>> >> some of the strange reports we have been getting.  Are
you
> >> >>> saying that
> >> >>> >> if we detect the cycle we default back to the way I am
doing
> >> >>> it now?
> >> >>> >> order within the groups...
> >> >>> >>
> >> >>> >> jesse
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --jesse mcconnell
> >> >>> >> jesse.mcconnell@gmail.com
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >>
> >
> >
> > --
> > jesse mcconnell
> > jesse.mcconnell@gmail.com
>


-- 
jesse mcconnell
jesse.mcconnell@gmail.com
Mime
View raw message