maven-repo-maintainers mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason van Zyl <jvan...@sonatype.com>
Subject Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing the Central repository
Date Fri, 28 Nov 2008 19:39:21 GMT
Yoav,

If we wanted to tarpit you or be more malicious that would be very  
easy for us to do. We could completely screw you over and cripple your  
access forever and there's not really much you could do about it if  
that was our motivation. Brian is completely open about what happened  
and actually being pretty forgiving. What you've done is essentially  
amounts to gross negligence.

You cost us very real amounts of money in wasted bandwidth.

You cost every Maven user real losses with respect to accessibility to  
the central repository.

To even insinuate that Brian is doing anything other then trying to  
protect the accessibility of central is pretty low.

I would be happy to accept your apology along with US20K that you've  
cost us in wasted bandwidth since your shoddily implemented strategy  
to grab the Nexus index. You've wasted everyone's time and money. Not  
appreciated. You'll be sorely mistaken if you think I'll stand by and  
watch you try to implicate Brian partaking in some plan to screw  
Artifactory over when that's exactly what Artifactory did to the Maven  
community. Take your lumps and fix it.

It's not Apache that pays for the resources, it's the Maven PMC as  
individuals with their contributions of time and money that keep it  
all working. Nothing is free, certainly not the machines, personnel  
and bandwidth required to keep central going. Brian acts on behalf of  
the Maven PMC not as the Nexus project manager and there isn't a  
single person on the Maven PMC who doesn't agree with him.

On 28-Nov-08, at 9:28 AM, Brian E. Fox wrote:

> Yoav,
> I sent the notice to everyone who might be affected by this as soon  
> as we understood this issue and confirmed it by selectively blocking  
> the suspected incidents (which was before 8am local time when I woke  
> up and saw that Contegix had confirmed it). Since it was clear that  
> users could directly affect this themselves by adjusting the cron  
> time, I thought it prudent to make that request ASAP. Waiting longer  
> serves no one in this case. Since the bandwidth costs real money,  
> and was causing users real problems, not blocking wasn't an option  
> once we understood how to fix it. Blocking and not notifying  
> everyone involved immediately wouldn't be in the user's best  
> interests either.
>
> If my motivation was other than to protect Central, we could have  
> more easily just blocked the high offending IPs and gone back to  
> enjoying the holiday quietly. Instead, we spent a lot of time really  
> understanding the problem to make sure we didn't needlessly take out  
> all Artifactory users in a way that actually broke builds.
>
> I have no doubt that the fix is easy, but it will take time to get  
> people running fixed versions. Adjusting cron can be done now and  
> will at least minimize the impact in the meantime so we can unblock  
> the index.
>
> --Brian
>
> -----Original Message-----
> From: Milos.Kleint@Sun.COM [mailto:Milos.Kleint@Sun.COM]
> Sent: Friday, November 28, 2008 12:01 PM
> To: Maven Users List
> Cc: artifactory-users@lists.sourceforge.net; repo-maintainers@maven.apache.org 
> ; Maven Project Management Committee List
> Subject: Re: [Artifactory-users] Artifactory 1.3 has been DOS'ing  
> the Central repository
>
> Yoav Landman wrote:
>> Brian,
>>
>> I think it is only a reasonable expectation from you, to contact  
>> our team
>> immediately as soon as you found out about this bug, so that we can  
>> let our
>> users know how to work around it and if they are affected at all.  
>> However,
>> since you are the Nexus commercial project manager, we are  
>> concerned that
>> the way you chose to handle this is not coming purely of sincere  
>> concern for
>> the Maven community.
>>
> Well, since the blocking started today, then obviously the culprit was
> found just lately. The performance of central repo was way bad  
> recently,
> so I agree with Brian that the blocking was necessary. He questioned  
> me
> about the Netbeans integration and how it download the nexus index  
> this
> week as well, so I don't think there are some hideous motivations  
> behind
> the move. I'm sure that once you fix the issue and the central hit
> flooding stops (means people using your betas upgrade to a version  
> that
> is fixed), artifactory clients will be reenabled.
>
> Regards
>
> Milos
>
>> This issue is only with the 1.3.0 *betas* (up to beta-6) and has 2  
>> simple
>> workarounds: Use an invalid CRON value or exclude repo1 from  
>> periodic index
>> downloads.
>> In any case, Artifactory will download the index on demand, which  
>> will use a
>> HEAD request first to test for unmodified index (sadly repo1 does  
>> not like
>> conditional GETs).
>>
>> We will easily fix this issue in the upcoming release.
>>
>> To disable the indexer cron set in the UI or in  
>> artifactory.config.xml file:
>> <indexer>
>>        <cronExp>invalid</cronExp>
>> </indexer>
>> And you should see in your log:
>> [ERROR] (o.a.r.index.
>> IndexerManagerImpl:72) Bad indexer cron expression 'invalid' will  
>> be ignored
>> (Illegal characters for this position: 'INV').
>>
>> To remove repo1 from periodic index downloads:
>>    <indexer>
>>        <cronExp>0 0 /1 * * *</cronExp>
>>        <excludedRepositories>
>>            <repositoryRef>repo1</repositoryRef>
>>        </excludedRepositories>
>>    </indexer>
>>
>> Happy Thanksgiving,
>>
>> Yoav Landman
>> The Artifactory Team
>>
>>
>> On Fri, Nov 28, 2008 at 2:50 PM, Brian E. Fox <brianf@reply.infinity.nu 
>> >wrote:
>>
>>
>>> Since approximately Mid August, the load on Central has been  
>>> growing at
>>> an exponential rate. You may have noticed slowdowns or dropped  
>>> connections
>>> recently as a side effect. We first had issues with Apache HTTPD  
>>> load
>>> increasing above the capacity of the machine. We switched over to  
>>> Nginx (
>>> http://blogs.sonatype.com/people/brian/2008/10/29/nginx-is-centrals-new-friend/)
>>> and this resolved the load, but then the 100mbps connection was  
>>> regularly
>>> becoming saturated. Every hour, on the hour for about 20 minutes  
>>> around the
>>> clock, the connection would max out and then return to about 50%
>>> utilization. We spent many days working with Contegix to diagnose  
>>> the
>>> problem but no single source stood out immediately.
>>>
>>>
>>>
>>> Yesterday we finally discovered that nearly all the traffic, both  
>>> the
>>> hourly spikes and the 50% background traffic is being caused by  
>>> downloads of
>>> the nexus-index.zip. After investigating the various tools that  
>>> use this
>>> data, we have concluded that Artifactory has a critical bug  
>>> ( apparently
>>> since June: http://issues.jfrog.org/jira/browse/RTFACT-390) that is
>>> causing every 1.3 instance to repeatedly download the 27mb zip  
>>> file. We
>>> found many cases of a single ip downloading the index more than  
>>> 1000 times a
>>> day! In the config it is set as follows:
>>>
>>>
>>>
>>>    <!-- The cron definition to control the activation of the  
>>> m2eclipse
>>> indexer. -->
>>>
>>>    <indexer>
>>>
>>>        <!-- By Default index every 5 hours -->
>>>
>>>        <cronExp>0 0 /5 * * ?</cronExp>
>>>
>>>    </indexer>
>>>
>>>
>>>
>>> (this is a quartz syntax which is "s m h…")
>>>
>>>
>>>
>>> This by itself wouldn't be a huge issue except for the fact that
>>> Artifactory ignores the index.properties file which contains the  
>>> last update
>>> timestamp AND doesn't first issue a HEAD to check the timestamp.  
>>> This means
>>> that every Artifactory 1.3 instance is grabbing this 27mb file at  
>>> least
>>> every 5 hours (we can't explain why certain ips are doing it 1000+  
>>> times a
>>> day..perhaps the config was modified there or some other  
>>> scheduling issue is
>>> present).
>>>
>>>
>>>
>>> For reference, the index on central is only updated once a week on  
>>> Sundays.
>>>
>>>
>>>
>>>
>>> To protect the Maven Community from ongoing troubles, we have had  
>>> to take
>>> the extra-ordinary step of blocking all downloads of the index  
>>> file by
>>> Artifactory until this is resolved. Upon doing this, the traffic  
>>> has fallen
>>> to 10% of what it has been in the recent past. If you are using  
>>> Artifactory,
>>> please adjust this cron definition to run only weekly and save  
>>> yourself and
>>> us tons of wasted bandwidth and money.
>>>
>>>
>>>
>>> Note that other tools like the Nexus Maven Repository Manager (
>>> http://nexus.sonatype.org), M2e and Q4e use the Nexus Indexer API  
>>> and are
>>> immune to this problem and are not blocked from downloading the  
>>> index. A new
>>> version of Nexus and the Nexus Indexer API will be published soon  
>>> along with
>>> M2e that will leverage incremental indexes to significantly reduce  
>>> the
>>> download requirements and allow near real time index updates.
>>>
>>>
>>>
>>> Brian Fox
>>>
>>> Apache Maven PMC
>>>
>>> http://blogs.sonatype.org/people/brian
>>>
>>> -------------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>> challenge
>>> Build the coolest Linux based applications with Moblin SDK & win  
>>> great
>>> prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in  
>>> the world
>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>> _______________________________________________
>>> Artifactory-users mailing list
>>> Artifactory-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>

Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder,  Apache Maven
jason at sonatype dot com
----------------------------------------------------------

Our achievements speak for themselves. What we have to keep track
of are our failures, discouragements and doubts. We tend to forget
the past difficulties, the many false starts, and the painful
groping. We see our past achievements as the end result of a
clean forward thrust, and our present difficulties as
signs of decline and decay.

  -- Eric Hoffer, Reflections on the Human Condition


Mime
View raw message