maven-repo-maintainers mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankostis <ankos...@gmail.com>
Subject Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central repository
Date Fri, 28 Nov 2008 18:52:42 GMT
Hi,

Although it is obvious Brian that you did an excellent debugging-job
with identifying the root of the DOS,
the final 'Note' at the end of your initial mail looked more like a
careless call for a flame-war:

On Fri 28 Nov 2008 14:50:17 Brian E. Fox wrote:
> ...
> Note that other tools like the Nexus Maven Repository Manager
> (http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API and
> are immune to this problem and are not blocked from downloading the
> index. A new version of Nexus and the Nexus Indexer API will be
> published soon along with M2e that will leverage incremental indexes to
> significantly reduce the download requirements and allow near real time
> index updates.
>
> Brian Fox
> Apache Maven PMC

* Why did you choose to compare Artifactory with other projects?
* Why did you have to supply the links to Nexus at this very moment?
* Why did you inform about the new version of Nexus coming-soon?

I hope that you do not hold any bad feeling for the people around
Artifactory just because they provided a decent maven-proxy when there
was none available?

I applaud both efforts, Nexus and Artifactory, (we are actually using
both!) but personally
i don't like anyone using its "maven-generated" status to enforce any
one project!

Such mind-sets are not fitted for the free-software programmers.


With Respect
  Kostis Anagnostopoulos



On Fri, Nov 28, 2008 at 7:28 PM, Brian E. Fox <brianf@reply.infinity.nu> wrote:
> Yoav,
> I sent the notice to everyone who might be affected by this as soon as we understood
this issue and confirmed it by selectively blocking the suspected incidents (which was before
8am local time when I woke up and saw that Contegix had confirmed it). Since it was clear
that users could directly affect this themselves by adjusting the cron time, I thought it
prudent to make that request ASAP. Waiting longer serves no one in this case. Since the bandwidth
costs real money, and was causing users real problems, not blocking wasn't an option once
we understood how to fix it. Blocking and not notifying everyone involved immediately wouldn't
be in the user's best interests either.
>
> If my motivation was other than to protect Central, we could have more easily just blocked
the high offending IPs and gone back to enjoying the holiday quietly. Instead, we spent a
lot of time really understanding the problem to make sure we didn't needlessly take out all
Artifactory users in a way that actually broke builds.
>
> I have no doubt that the fix is easy, but it will take time to get people running fixed
versions. Adjusting cron can be done now and will at least minimize the impact in the meantime
so we can unblock the index.
>
> --Brian
>
> -----Original Message-----
> From: Milos.Kleint@Sun.COM [mailto:Milos.Kleint@Sun.COM]
> Sent: Friday, November 28, 2008 12:01 PM
> To: Maven Users List
> Cc: artifactory-users@lists.sourceforge.net; repo-maintainers@maven.apache.org; Maven
Project Management Committee List
> Subject: Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central repository
>
> Yoav Landman wrote:
>> Brian,
>>
>> I think it is only a reasonable expectation from you, to contact our team
>> immediately as soon as you found out about this bug, so that we can let our
>> users know how to work around it and if they are affected at all. However,
>> since you are the Nexus commercial project manager, we are concerned that
>> the way you chose to handle this is not coming purely of sincere concern for
>> the Maven community.
>>
> Well, since the blocking started today, then obviously the culprit was
> found just lately. The performance of central repo was way bad recently,
> so I agree with Brian that the blocking was necessary. He questioned me
> about the Netbeans integration and how it download the nexus index this
> week as well, so I don't think there are some hideous motivations behind
> the move. I'm sure that once you fix the issue and the central hit
> flooding stops (means people using your betas upgrade to a version that
> is fixed), artifactory clients will be reenabled.
>
> Regards
>
> Milos
>
>> This issue is only with the 1.3.0 *betas* (up to beta-6) and has 2 simple
>> workarounds: Use an invalid CRON value or exclude repo1 from periodic index
>> downloads.
>> In any case, Artifactory will download the index on demand, which will use a
>> HEAD request first to test for unmodified index (sadly repo1 does not like
>> conditional GETs).
>>
>> We will easily fix this issue in the upcoming release.
>>
>> To disable the indexer cron set in the UI or in artifactory.config.xml file:
>> <indexer>
>>         <cronExp>invalid</cronExp>
>> </indexer>
>> And you should see in your log:
>> [ERROR] (o.a.r.index.
>> IndexerManagerImpl:72) Bad indexer cron expression 'invalid' will be ignored
>> (Illegal characters for this position: 'INV').
>>
>> To remove repo1 from periodic index downloads:
>>     <indexer>
>>         <cronExp>0 0 /1 * * *</cronExp>
>>         <excludedRepositories>
>>             <repositoryRef>repo1</repositoryRef>
>>         </excludedRepositories>
>>     </indexer>
>>
>> Happy Thanksgiving,
>>
>> Yoav Landman
>> The Artifactory Team
>>
>>
>> On Fri, Nov 28, 2008 at 2:50 PM, Brian E. Fox <brianf@reply.infinity.nu>wrote:
>>
>>
>>>  Since approximately Mid August, the load on Central has been growing at
>>> an exponential rate. You may have noticed slowdowns or dropped connections
>>> recently as a side effect. We first had issues with Apache HTTPD load
>>> increasing above the capacity of the machine. We switched over to Nginx (
>>> http://blogs.sonatype.com/people/brian/2008/10/29/nginx-is-centrals-new-friend/)
>>> and this resolved the load, but then the 100mbps connection was regularly
>>> becoming saturated. Every hour, on the hour for about 20 minutes around the
>>> clock, the connection would max out and then return to about 50%
>>> utilization. We spent many days working with Contegix to diagnose the
>>> problem but no single source stood out immediately.
>>>
>>>
>>>
>>> Yesterday we finally discovered that nearly all the traffic, both the
>>> hourly spikes and the 50% background traffic is being caused by downloads of
>>> the nexus-index.zip. After investigating the various tools that use this
>>> data, we have concluded that Artifactory has a critical bug ( apparently
>>> since June: http://issues.jfrog.org/jira/browse/RTFACT-390) that is
>>> causing every 1.3 instance to repeatedly download the 27mb zip file. We
>>> found many cases of a single ip downloading the index more than 1000 times a
>>> day! In the config it is set as follows:
>>>
>>>
>>>
>>>     <!-- The cron definition to control the activation of the m2eclipse
>>> indexer. -->
>>>
>>>     <indexer>
>>>
>>>         <!-- By Default index every 5 hours -->
>>>
>>>         <cronExp>0 0 /5 * * ?</cronExp>
>>>
>>>     </indexer>
>>>
>>>
>>>
>>> (this is a quartz syntax which is "s m h…")
>>>
>>>
>>>
>>> This by itself wouldn't be a huge issue except for the fact that
>>> Artifactory ignores the index.properties file which contains the last update
>>> timestamp AND doesn't first issue a HEAD to check the timestamp. This means
>>> that every Artifactory 1.3 instance is grabbing this 27mb file at least
>>> every 5 hours (we can't explain why certain ips are doing it 1000+ times a
>>> day..perhaps the config was modified there or some other scheduling issue is
>>> present).
>>>
>>>
>>>
>>> For reference, the index on central is only updated once a week on Sundays.
>>>
>>>
>>>
>>>
>>> To protect the Maven Community from ongoing troubles, we have had to take
>>> the extra-ordinary step of blocking all downloads of the index file by
>>> Artifactory until this is resolved. Upon doing this, the traffic has fallen
>>> to 10% of what it has been in the recent past. If you are using Artifactory,
>>> please adjust this cron definition to run only weekly and save yourself and
>>> us tons of wasted bandwidth and money.
>>>
>>>
>>>
>>> Note that other tools like the Nexus Maven Repository Manager (
>>> http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API and are
>>> immune to this problem and are not blocked from downloading the index. A new
>>> version of Nexus and the Nexus Indexer API will be published soon along with
>>> M2e that will leverage incremental indexes to significantly reduce the
>>> download requirements and allow near real time index updates.
>>>
>>>
>>>
>>> Brian Fox
>>>
>>> Apache Maven PMC
>>>
>>> http://blogs.sonatype.org/people/brian
>>>
>>> -------------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>> challenge
>>> Build the coolest Linux based applications with Moblin SDK & win great
>>> prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>> _______________________________________________
>>> Artifactory-users mailing list
>>> Artifactory-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>

Mime
View raw message