archiva-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stallard,David" <stall...@oclc.org>
Subject RE: 100% CPU in Archiva 1.3.5
Date Fri, 28 Oct 2011 18:24:13 GMT
Here's the latest update.  Throughout the day Thursday, the indexing
queue gradually climed from 19,000 to 23,000.  I figured that maybe
after hours when there's less activity, it would make headway.  But
Friday morning, the queue was at 27,000.

I had to bounce it because some users were reporting 405 errors when
trying to upload to Archiva.  After the bounce, the queue was very
small...it started at 3 and slowly climbed, it was at 898 when the
hourly indexes kicked in.  The bounce did seem to fix the 405 errors
though.

But then this afternoon, we got reports of 405 errors again.  I bounced,
and the queue was at 0!  During this time, CPU usage was normal...the
archiva java process was using trivial CPU.  But then the queue went up
to 3, and now we have 100% CPU usage again.

So it appears that ANY indexing queue at all causes us to hit 100% CPU.
The queue is currently at 7...I'm not sure how often it actually is
empty.

I see that 1.4-M1 was released a few days ago...does it include anything
that might address this?
 

-----Original Message-----
From: Stallard,David 
Sent: Thursday, October 27, 2011 9:24 AM
To: 'users@archiva.apache.org'
Subject: RE: 100% CPU in Archiva 1.3.5

To follow up on my previous message...you mentioned that frequent
deployments could create the backlog.  We do have a build server which
is constantly doing builds of many different products, and each build
uploads snapshots to Archiva, so my guess would be that maybe we had a
period of heavier than normal builds so that Archiva was inundated with
new snapshots to index.

-----Original Message-----
From: Stallard,David
Sent: Thursday, October 27, 2011 9:17 AM
To: 'users@archiva.apache.org'
Subject: RE: 100% CPU in Archiva 1.3.5

Thanks, Brett.  This morning our indexing queue is around 19,000 instead
of 135,000, so it must be getting close to catching up with the
indexing.  I'm not sure how we got to having such a huge backlog; we
pinpointed the day and hour that the CPU usage suddenly increased, and
in the logs it doesn't look like we had a lot of new files uploaded
around that particular time.  Maybe at some point the indexing stopped
for some reason and then later it started up again?  I don't even know
if that's possible, just thinking aloud.

-----Original Message-----
From: Brett Porter [mailto:brett@porterclan.net] On Behalf Of Brett
Porter
Sent: Thursday, October 27, 2011 1:39 AM
To: users@archiva.apache.org
Subject: Re: 100% CPU in Archiva 1.3.5

It's working through a backlog of indexing operations, which are CPU and
disk intensive. These may have been there from a scan, or from frequent
deployments.

Keep an eye on whether that count stays high over time. If so, you may
need to prevent scanning certain repositories or artifact types that you
don't need searched.

- Brett

On 27/10/2011, at 12:01 AM, Stallard,David wrote:

> Although the CPU is still at 100%, Archiva has actually been 
> responsive for the last 24 hours.  Here's what the System Status page 
> looks like at the moment:
> 
> Queues
> Queue Size
> database-update 0
> indexing 135310
> repository-scanning 0
> 
> Repository Scans Currently in Progress No scans in progress.
> 
> Caches
> Cache Size Hits Misses Hit Ratio   
> operations 25 11 50 18% Flush
> users 5 130523 106 100% Flush
> resources 2 2 4 33% Flush
> url-failures-cache 1000 0 0 0% Flush
> userPermissions 2 132801 711 99% Flush effective-project-cache 1000
> 87048 30925 74% Flush roles 12 308 29 91% Flush permissions 31 926 62 
> 94% Flush effectiveRoleSet 0 0 0 0% Flush userAssignments 0 2 4 33% 
> Flush ehcache 0 0 0 0% Flush keys 80 5 371 1% Flush
> 
> Memory Usage
> 291M/999M (Max: 999M)
> 
> Current Time
> Wed Oct 26 08:59:06 EDT 2011
> 
> -----Original Message-----
> From: Brett Porter [mailto:brett@porterclan.net] On Behalf Of Brett 
> Porter
> Sent: Wednesday, October 26, 2011 1:33 AM
> To: users@archiva.apache.org
> Subject: Re: 100% CPU in Archiva 1.3.5
> 
> Are you able to visit the "system status" page before it becomes 
> unresponsive and see what is in progress, and the status of the heap?
> 
> - Brett
> 
> On 26/10/2011, at 7:48 AM, Stallard,David wrote:
> 
>> We have been running Archiva 1.3.4 for several months without issue.
>> Last week, CPU usage shot up to 100% or higher...we saw it go as high

>> as 300% (this is a 4-CPU host).  We were not able to tie this to any 
>> particular event.
>> 
>> I discovered that 1.3.5 fixes a Dependency Tree issue which can cause

>> high CPU usage, so yesterday we upgraded to 1.3.5.  At first the 
>> problem seemed resolved, but over the next 15 hours the CPU usage 
>> continued to increase until now we are back to having CPU usage at
> 100% consistently.
>> When the hourly scans of internal and snapshots happen, this jumps up

>> to around 200%.
>> 
>> This morning Archiva became unresponsive, which is when I discovered 
>> that the high CPU is still an issue.  I bounced Archiva; while this 
>> made the system responsive again, the CPU was immediately back at 
>> 100%
> usage.
>> 
>> The logs show no unusual activity.
>> 
>> Any idea what might be happening?
>> 
>> Thanks,
>> David
> 
> --
> Brett Porter
> brett@apache.org
> http://brettporter.wordpress.com/
> http://au.linkedin.com/in/brettporter
> 
> 
> 
> 
> 
> 

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter







Mime
View raw message