continuum-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Porter <br...@apache.org>
Subject Re: build scheduling issues
Date Fri, 10 Nov 2006 00:35:17 GMT
I'm lost :)

My point below was that if you just took all the projects and sorted  
them, there is no cycle in the example you gave. The cycle would come  
if you tried to order them by groups, because there are dependencies  
between projects within the groups in both directions.

so...
In 1.0.3, scheduling a project didn't build any dependencies, but if  
you scheduled a bunch then they were built in order. I don't know  
what happened if there were cycles.
In trunk, it does what is below, or is that where you want to go with  
it? And building dependencies is a next step on top of that?

Honestly, I'm not seeing where cycles are going to be a common case,  
and when they occur I don't think we need to be too concerned about  
the order they build in. I have been extremely dense recently though  
so I'm sure I'm missing something :)

Cheers,
Brett

On 10/11/2006, at 2:20 AM, Jesse McConnell wrote:

> sorry, once changed how?  once the group dependencies are sorted out?
>
> if we are taking all projects across all groups and trying to generate
> a cycle free graph then cycles are potentially a problem.
>
> just so we are clear in my mind:
>
> * grab all projects across all groups
> * attempt to sort
> * if success, build normally
> * if failure, notify of cycle and proceed to sequentially process
> groups, sorting projects in that group only and building
> * if cycle detected in that group then build all projects in that
> group in any old order that comes out of the database
>
> The difference between this and what used to exist before my group
> changes is that it will attempt to order inside the group and build as
> opposed to before where it would just failover to building all
> projects in whatever order came from the db.
>
> is that where we are with this?
>
> jesse
>
>
>
> On 11/8/06, Brett Porter <brett@apache.org> wrote:
>> That's a cycle between the groups, not the projects, so shouldn't be
>> a problem once changed.
>>
>> - Brett
>>
>> On 09/11/2006, at 1:57 AM, Jesse McConnell wrote:
>>
>> > the issue isn't cycles within a normal m2 build, but when you  
>> shove 4
>> > or 5 of them together into one continuum instance and then ask them
>> > all to play together.
>> >
>> > For example if you shoved continuum, maven-shared and archiva  
>> all into
>> > the same continuum instance and triggered a build with all of  
>> them in
>> > the same directed graph then we have had instances of cycles in the
>> > past (not sure about right now, mpir cycle might be refactored out
>> > thanks to joakim).
>> >
>> > emm had another example last night when we were talking about it as
>> > well.
>> >
>> > jesse
>> >
>> > On 11/8/06, Brett Porter <brett@apache.org> wrote:
>> >> Sorry, am I missing something? How do we end up with a cycle in
>> >> Continuum? Is there a specific example?
>> >>
>> >> I know it's possible - we had to allow it in the repository (eg,
>> >> dom4j <-> jaxen), but it is certainly undesirable and honestly  
>> should
>> >> be rare, especially in m2 built artifacts. Should it produce a  
>> build
>> >> warning?
>> >>
>> >> Basically, the treatment there is to just stop following the tree
>> >> when you hit the cycle, rather than changing the way things are
>> >> ordered. So it's really arbitrary which might come first, but  
>> isn't
>> >> really a concern there.
>> >>
>> >> Anyway, just wondering.
>> >>
>> >> Cheers,
>> >> Brett
>> >>
>> >> On 08/11/2006, at 11:46 PM, Emmanuel Venisse wrote:
>> >>
>> >> > Yes
>> >> >
>> >> > Jesse McConnell a écrit :
>> >> >> we should add a page that analyzes each schedule for  
>> cycles...that
>> >> >> would be a cool little feature
>> >> >> On 11/8/06, Emmanuel Venisse <emmanuel@venisse.net> wrote:
>> >> >>> Yes, we need a global ordering, so projects will be build
>> >> >>> independently of groups, because in some
>> >> >>> case a cycle can be created between groups (not necessary 

>> between
>> >> >>> projects).
>> >> >>>
>> >> >>> In case a cycle is detected between projects, continuum can't
>> >> >>> find the build order. In this case, I
>> >> >>> think we need to sort a little project so will reduce build
>> >> >>> errors. So if we have a cycle, we can
>> >> >>> sort projects in a group and build them. In most of case  
>> (maven
>> >> >>> projects), we don't have a cycle in
>> >> >>> a group.
>> >> >>>
>> >> >>> Emmanuel
>> >> >>>
>> >> >>> Brett Porter a écrit :
>> >> >>> > I think you want global ordering. Grouping should just
be a
>> >> >>> > display/management technique, not anything that changes
how
>> >> >>> projects are
>> >> >>> > handled.
>> >> >>> >
>> >> >>> > However, this needs to be reviewed as a whole (which I
think
>> >> >>> Emmanuel is
>> >> >>> > doing), such that builds can be triggered when their
>> >> >>> dependencies change
>> >> >>> > which will help with the ordering as it won't be  
>> dependant on
>> >> >>> them all
>> >> >>> > being triggered at the same time?
>> >> >>> >
>> >> >>> > - Brett
>> >> >>> >
>> >> >>> > On 08/11/2006, at 9:51 AM, Jesse McConnell wrote:
>> >> >>> >
>> >> >>> >> I was reading through the DefaultContinuum.buildProjects
>> >> >>> ( Schedule id
>> >> >>> >> ) method and after discussing some things with Emmanuel...I
>> >> >>> think we
>> >> >>> >> have a problem here.  When I went through and refactored
>> >> >>> things to
>> >> >>> >> support a more Project Group centric setup with continuum
I
>> >> >>> changed
>> >> >>> >> this method a bit.
>> >> >>> >>
>> >> >>> >> Originally, this method would gather up all projects
that
>> >> >>> would be
>> >> >>> >> triggered by that schedule, run them all through the
 
>> project
>> >> >>> sorter
>> >> >>> >> and then build each in sequence.
>> >> >>> >>
>> >> >>> >> When I added the project groups to this mix, I changed
 
>> things
>> >> >>> to be on
>> >> >>> >> a project group basis, so that on a project group
by  
>> project
>> >> >>> group
>> >> >>> >> basis it would order the projects and build them.
 At the
>> >> time I
>> >> >>> >> thought this was the way to go...but maybe not.
>> >> >>> >>
>> >> >>> >> 17:14 <evenisse> we need to take all projects
from all  
>> groups,
>> >> >>> sort them
>> >> >>> >> 17:15 <evenisse> if we don't have a cycle, it's
ok and we
>> >> >>> build all
>> >> >>> >> 17:15 <evenisse> if it isn't ok, we sort project
by group
>> >> >>> >>
>> >> >>> >> For example, if we loaded up a Plexus group and a
Maven
>> >> >>> group...the
>> >> >>> >> way it currently is (with my change) it would process
all
>> >> >>> triggered
>> >> >>> >> builds within one group and then process all triggered
 
>> builds
>> >> >>> in the
>> >> >>> >> other group.   This would not take into account potential
>> >> >>> dependencies
>> >> >>> >> between the two.
>> >> >>> >>
>> >> >>> >> Does anyone have any thoughts on this?  I am inclined
to  
>> fix
>> >> >>> it up so
>> >> >>> >> its like it used to be where all projects across all
 
>> project
>> >> >>> groups
>> >> >>> >> are thrown into the graph....I keep feeling like I
am  
>> missing
>> >> >>> >> something wrong with this, but I can't pin it down.
>> >> >>> >>
>> >> >>> >> One thing that perhaps Emmanuel could explain a bit
more is
>> >> >>> the third
>> >> >>> >> comment there.  In our conversation on this he said
that he
>> >> >>> thinks
>> >> >>> >> that the cycles are cropping up all the time, and
if  
>> thats the
>> >> >>> case
>> >> >>> >> then we are building a lot of unordered builds which
would
>> >> >>> account for
>> >> >>> >> some of the strange reports we have been getting.
 Are you
>> >> >>> saying that
>> >> >>> >> if we detect the cycle we default back to the way
I am  
>> doing
>> >> >>> it now?
>> >> >>> >> order within the groups...
>> >> >>> >>
>> >> >>> >> jesse
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> --jesse mcconnell
>> >> >>> >> jesse.mcconnell@gmail.com
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >> >>>
>> >>
>> >
>> >
>> > --
>> > jesse mcconnell
>> > jesse.mcconnell@gmail.com
>>
>
>
> -- 
> jesse mcconnell
> jesse.mcconnell@gmail.com

Mime
View raw message