maven-repo-maintainers mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian E. Fox" <>
Subject RE: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central repository
Date Fri, 28 Nov 2008 17:28:21 GMT
I sent the notice to everyone who might be affected by this as soon as we understood this
issue and confirmed it by selectively blocking the suspected incidents (which was before 8am
local time when I woke up and saw that Contegix had confirmed it). Since it was clear that
users could directly affect this themselves by adjusting the cron time, I thought it prudent
to make that request ASAP. Waiting longer serves no one in this case. Since the bandwidth
costs real money, and was causing users real problems, not blocking wasn't an option once
we understood how to fix it. Blocking and not notifying everyone involved immediately wouldn't
be in the user's best interests either.

If my motivation was other than to protect Central, we could have more easily just blocked
the high offending IPs and gone back to enjoying the holiday quietly. Instead, we spent a
lot of time really understanding the problem to make sure we didn't needlessly take out all
Artifactory users in a way that actually broke builds.

I have no doubt that the fix is easy, but it will take time to get people running fixed versions.
Adjusting cron can be done now and will at least minimize the impact in the meantime so we
can unblock the index.


-----Original Message-----
From: Milos.Kleint@Sun.COM [mailto:Milos.Kleint@Sun.COM] 
Sent: Friday, November 28, 2008 12:01 PM
To: Maven Users List
Cc:;; Maven Project
Management Committee List
Subject: Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central repository

Yoav Landman wrote:
> Brian,
> I think it is only a reasonable expectation from you, to contact our team
> immediately as soon as you found out about this bug, so that we can let our
> users know how to work around it and if they are affected at all. However,
> since you are the Nexus commercial project manager, we are concerned that
> the way you chose to handle this is not coming purely of sincere concern for
> the Maven community.
Well, since the blocking started today, then obviously the culprit was 
found just lately. The performance of central repo was way bad recently, 
so I agree with Brian that the blocking was necessary. He questioned me 
about the Netbeans integration and how it download the nexus index this 
week as well, so I don't think there are some hideous motivations behind 
the move. I'm sure that once you fix the issue and the central hit 
flooding stops (means people using your betas upgrade to a version that 
is fixed), artifactory clients will be reenabled.



> This issue is only with the 1.3.0 *betas* (up to beta-6) and has 2 simple
> workarounds: Use an invalid CRON value or exclude repo1 from periodic index
> downloads.
> In any case, Artifactory will download the index on demand, which will use a
> HEAD request first to test for unmodified index (sadly repo1 does not like
> conditional GETs).
> We will easily fix this issue in the upcoming release.
> To disable the indexer cron set in the UI or in artifactory.config.xml file:
> <indexer>
>         <cronExp>invalid</cronExp>
> </indexer>
> And you should see in your log:
> [ERROR] (o.a.r.index.
> IndexerManagerImpl:72) Bad indexer cron expression 'invalid' will be ignored
> (Illegal characters for this position: 'INV').
> To remove repo1 from periodic index downloads:
>     <indexer>
>         <cronExp>0 0 /1 * * *</cronExp>
>         <excludedRepositories>
>             <repositoryRef>repo1</repositoryRef>
>         </excludedRepositories>
>     </indexer>
> Happy Thanksgiving,
> Yoav Landman
> The Artifactory Team
> On Fri, Nov 28, 2008 at 2:50 PM, Brian E. Fox <>wrote:
>>  Since approximately Mid August, the load on Central has been growing at
>> an exponential rate. You may have noticed slowdowns or dropped connections
>> recently as a side effect. We first had issues with Apache HTTPD load
>> increasing above the capacity of the machine. We switched over to Nginx (
>> and this resolved the load, but then the 100mbps connection was regularly
>> becoming saturated. Every hour, on the hour for about 20 minutes around the
>> clock, the connection would max out and then return to about 50%
>> utilization. We spent many days working with Contegix to diagnose the
>> problem but no single source stood out immediately.
>> Yesterday we finally discovered that nearly all the traffic, both the
>> hourly spikes and the 50% background traffic is being caused by downloads of
>> the After investigating the various tools that use this
>> data, we have concluded that Artifactory has a critical bug ( apparently
>> since June: that is
>> causing every 1.3 instance to repeatedly download the 27mb zip file. We
>> found many cases of a single ip downloading the index more than 1000 times a
>> day! In the config it is set as follows:
>>     <!-- The cron definition to control the activation of the m2eclipse
>> indexer. -->
>>     <indexer>
>>         <!-- By Default index every 5 hours -->
>>         <cronExp>0 0 /5 * * ?</cronExp>
>>     </indexer>
>> (this is a quartz syntax which is "s m h…")
>> This by itself wouldn't be a huge issue except for the fact that
>> Artifactory ignores the file which contains the last update
>> timestamp AND doesn't first issue a HEAD to check the timestamp. This means
>> that every Artifactory 1.3 instance is grabbing this 27mb file at least
>> every 5 hours (we can't explain why certain ips are doing it 1000+ times a
>> day..perhaps the config was modified there or some other scheduling issue is
>> present).
>> For reference, the index on central is only updated once a week on Sundays.
>> To protect the Maven Community from ongoing troubles, we have had to take
>> the extra-ordinary step of blocking all downloads of the index file by
>> Artifactory until this is resolved. Upon doing this, the traffic has fallen
>> to 10% of what it has been in the recent past. If you are using Artifactory,
>> please adjust this cron definition to run only weekly and save yourself and
>> us tons of wasted bandwidth and money.
>> Note that other tools like the Nexus Maven Repository Manager (
>>, M2e and Q4e use the Nexus Indexer API and are
>> immune to this problem and are not blocked from downloading the index. A new
>> version of Nexus and the Nexus Indexer API will be published soon along with
>> M2e that will leverage incremental indexes to significantly reduce the
>> download requirements and allow near real time index updates.
>> Brian Fox
>> Apache Maven PMC
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> _______________________________________________
>> Artifactory-users mailing list

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message