continuum-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Edward Gruber <cgru...@israfil.net>
Subject Re: Continuum queuing jobs but not performing builds
Date Fri, 05 Sep 2008 16:00:21 GMT
Damn good catch, Peter.

Christian.

On 5-Sep-08, at 11:49 , Peter Janes wrote:

> After having this happen a third time I took a closer look at the  
> database and saw that one of the entries in the BUILDRESULT table  
> had an END_TIME of 0.  When I changed that field to a current time  
> and restarted the server Continuum started to work again.
>
> I was able to reproduce the problem by setting the END_TIME of the  
> last BUILDRESULT to 0, starting the server, and waiting for that  
> project to be queued.  It was resolved again when I reset END_TIME  
> to its original value.
>
> I've reported this as http://jira.codehaus.org/browse/CONTINUUM-1871  
> and attached the thread dump.
>
> Thanks,
> Peter J.
>
> Brett Porter wrote:
>> I certainly haven't seen something like this - perhaps something in  
>> the
>> project change caused an infinite loop that might be a bug in the  
>> Maven
>> project handling.
>> Are you able to get a thread dump from Continuum when it occurs?
>> Thanks,
>> Brett
>> 2008/9/3 Peter Janes <continuum.apache.org@peterjanes.ca>
>>> Since adding a set of projects to a Continuum 1.2 instance last  
>>> Thursday,
>>> the server has stopped performing any builds on any projects.  The  
>>> server
>>> had performed hundreds of builds without issue over a period of  
>>> several
>>> months.
>>>
>>> The initial problem that appeared was that projects would be  
>>> queued but
>>> never started; as soon as the first job entered the queue  
>>> Continuum would
>>> start using 100% CPU and never start building anything.  Even when  
>>> I removed
>>> all of the projects from the queue it remained at 100% CPU.   
>>> Restarting the
>>> server didn't help: as soon as a job was queued the CPU would  
>>> spike and no
>>> further progress would be made.
>>>
>>> Thinking that somehow the (MySQL) database might be corrupt, I  
>>> removed it
>>> and started fresh.  I added the projects that appeared to cause  
>>> the problem
>>> and they built successfully, so I started re-adding the rest of the
>>> projects.  Unfortunately, at some point while I was doing this  
>>> builds
>>> stopped being executed again: this time they get queued up and are
>>> apparently started, but I don't see any build processes being  
>>> invoked.  Even
>>> the simplest pom-only job times out.  The timeouts are shown in the
>>> Continuum log but aren't being recorded in the database as  
>>> failures or
>>> errors, and restarting doesn't make a difference, which suggests  
>>> to me that
>>> perhaps the database has been corrupted again.
>>>
>>> I don't see anything in Jira or the list archives; has anyone seen
>>> behaviours like this?  More importantly, has anyone been able to  
>>> solve them?
>>>
>>> Environment: RHEL 5, Continuum 1.2 (built in May), Java 1.6.0_03,  
>>> MySQL
>>> 5.0.45, Maven 2.0.9
>>>
>>> Peter J.
>>> --
>>> Sometimes the Universe needs a change of perspective.
>>> --J. Michael Straczynski
>>>
>
>
> -- 
> Sometimes the Universe needs a change of perspective.
>  --J. Michael Straczynski
>


Mime
View raw message