archiva-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stallard,David" <stall...@oclc.org>
Subject Archiva becoming unresponsive nightly
Date Wed, 27 Nov 2013 16:18:35 GMT
Our Archiva 1.3.5 instance has started becoming unresponsive almost every night.  We have a
cron job which does a wget every 5 minutes on the main Archiva page to ensure that it's available.
 This cron job sends an alert if the wget fails and Archiva is then bounced.  The bounce does
make Archiva available again until the next night.

We suspect that the repository scans may be the cause, but we aren't sure.  We have the repository
scans (two of them, internal and snapshots) set to run nightly at 7pm.  They run sequentially,
so the last one finishes at roughly 11:30pm.  Then we have the database scan set to run at
3am.

Software builds that occur later in the night seem to hang when trying to upload a pom.  Here's
an example schedule of events:

1:00am - build starts
1:01am - build tries to upload a pom to Archiva and appears to hang waiting for the upload
to finish
1:45am - the wget sends an alarm and Archiva is bounced. At this point, the build resumes
but fails since the upload failed.

Since the wget doesn't alarm until 1:45am, this leads us to believe that Archiva may be degrading
in performance over time - it must be "alive" enough for the wget check to succeed, but "dead"
enough that uploads can't complete.  Here is the exact wget command our cron monitor is running:

                wget -tries 10 -q -O - http://our.archiva.url:port/archiva

There is nothing in archiva.log to indicate any problem.  In the example above, the log is
"business as usual" until roughly 1am, and then there are no other messages until Archiva
gets bounced, at which point we see the normal logging messages showing that Archiva is starting
up.

The problem occurred every day last week, but then Sunday and Monday everything was fine.
 The problem resumed on Tuesday night, but exactly 1 hour later than when it usually occurs.

A few weeks ago we deleted the index and database to let Archiva create them fresh, and this
eliminated a large number of errors which we had been getting on a daily basis.  So the index
and database are probably in better shape now than they'd been in the past.

We've looked to see if something external to Archiva, such as system backups, might be tying
up resources and affecting Archiva performance.  We've not found anything that seems like
a suspect.

Our primary theory at this point is that the repository scans are somehow adversely affecting
Archiva and at some point 1-2 hours after they complete, Archiva becomes unresponsive.  We
are setting up some test builds to run over the holiday weekend which will hopefully give
us more data to prove or disprove this (we are running a build every hour so we can better
pinpoint exactly when uploads stop working).

Any ideas?

As a workaround, based on the assumption that the scans are causing the problem, we have considered
changing the respository scan from daily to weekly.  Would this affect any aspect of Archiva
other than the ability to search for recently uploaded artifacts?

Thanks,
David

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message