maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Fox <bri...@infinity.nu>
Subject Re: Maven2 mirror @ netcologne
Date Wed, 25 Jul 2012 20:32:14 GMT
On Wed, Jul 25, 2012 at 4:19 PM, Brett Porter <brett@apache.org> wrote:

> On 26/07/2012, at 3:46 AM, Brian Fox wrote:
>
> > On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter <brett@apache.org> wrote:
> >
> >>
> >> My understanding is that unfortunately Sonatype are not allowing anyone
> >> else to mirror the content directly any more.
> >>
> >
> > Ibiblio disabled the rsync on their own accord because it was thrashing
> > their disks.
>
> That doesn't seem consistent with Joel's response on the ticket. When did
> that happen?
>
> If this is the case, we should establish another "escrow" location.
>
>
>

Let me clarify my statement because it's not opposed to what Joel said,
we're talking about different things. Ibiblio turned off OUTBOUND rsyncs
because too many people where thrashing their disks and chewing up massive
amounts of bandwidth. The INBOUND rsync is still happenning. Because
Central is over 600GB now, even the rsync from us was thrashing both of our
disks so Joel developed a new method that works like this:

Central Staging reaches out to all the forges and does mini rsyncs scoped
by each groupId folder so they aren't causing any real load. We take the
log output from those to get a manifest of the files that changed. They get
added all up and then inserted into the include-files portion of the
outbound rsync to the other central machines and to ibiblio. The net effect
is that it reduces the churn that rsync produces on the systems, speeds up
the sync and still keeps every file updated.

Since the outbound sync doesn't touch that last_updated.txt file, it isn't
being added as a changed file to any of the rsync pushes, hence it appears
outdated on Ibiblio. But if you poke around out there, you'll find that the
components are in fact being updated.

With the move to the CDN, we have moved a massive amount of traffic back
off of Ibiblio because they were previously serving the indexes. I'll ping
them and see if the rsync is possible again.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message