Mailing-List: contact dev-help@archiva.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@archiva.apache.org
Received-SPF: pass (athena.apache.org: domain of oching@exist.com designates
 209.85.198.230 as permitted sender)
Message-ID: <1a57a2980807160156t240039f0gede90e8632c867b5@mail.gmail.com>
Date: Wed, 16 Jul 2008 16:56:05 +0800
From: "Maria Odea Ching" <oching@apache.org>
Sender: oching@exist.com
To: dev@archiva.apache.org
Subject: Re: wrong metadatas
In-Reply-To: <f345f3480807160147m74211ca4gbd89e5bd9a3db4ef@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_11164_7815718.1216198566184"
References: <4c39e3030807090545u1bbc192fw3e4189a48b0d1086@mail.gmail.com>
	 <1a57a2980807100247n72066c89jf18a8c95de5fe23@mail.gmail.com>
	 <4c39e3030807100314x447cadbdl4b6bc33d0486eef0@mail.gmail.com>
	 <1a57a2980807100344j66eae939mcc095ceab218f805@mail.gmail.com>
	 <4c39e3030807110335g7807d828y6ddea2ea6d69d44d@mail.gmail.com>
	 <1a57a2980807110359u5e9eabedhd869d00786f9529e@mail.gmail.com>
	 <1a57a2980807130708g2037722fs8918033157dea91c@mail.gmail.com>
	 <1a57a2980807140225r547b9bcfs3ef25c0be122520f@mail.gmail.com>
	 <4c39e3030807160125j1a04b1e5nc8e336ed3a7496cf@mail.gmail.com>
	 <f345f3480807160147m74211ca4gbd89e5bd9a3db4ef@mail.gmail.com>

------=_Part_11164_7815718.1216198566184
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Yay! Thanks Nico!
I'll start the release now.. There were already a couple of issues that got
included in 1.1 while waiting for the blocker fixes :)

Thanks,
Deng

On Wed, Jul 16, 2008 at 4:47 PM, Emmanuel Venisse <
emmanuel.venisse@gmail.com> wrote:

> If it's ok, it is possible to restart the release or do you want to add
> more
> things in 1.1?
>
> Emmanuel
>
> On Wed, Jul 16, 2008 at 10:25 AM, nicolas de loof <nicolas@apache.org>
> wrote:
>
> > I made some tests and this works nicely for me.
> >
> > Thanks a lot !
> >
> > 2008/7/14 Maria Odea Ching <oching@apache.org>:
> >
> > > Additional fixes in -r676516 and -r676526..
> > > Summary of the fixes:
> > >
> > >   1. generate checksums for the generated merged metadata
> > >   2. put the merged metadata & its checksums in the first repo where
> the
> > >   metadata was found. the filename for the merged metadata is
> > >   maven-metadata-${repoGroupId}.xml to handle instances when a
> repository
> > >   belongs to multiple groups.
> > >   3. merge the metadata if and only if all the following conditions are
> > >   met:
> > >      - the request was made to a repository group url
> > >      - the requested resource is a metadata file or a checksum file of
> a
> > >      metadata
> > >      - the requested metadata file is at the project level, not version
> > >      level
> > >   4. added test case when metadata checksums are requested after the
> > >   metadata file (simulate Maven requests)
> > >
> > > Thanks,
> > > Deng
> > >
> > > On Sun, Jul 13, 2008 at 10:08 PM, Maria Odea Ching <oching@apache.org>
> > > wrote:
> > >
> > > > Hi Nico,
> > > >
> > > > Could you verify the fix i committed in trunk -r676322 if it works
> for
> > > you?
> > > > I tested it out  using ranges and I was able to get the artifacts
> from
> > > the
> > > > different repos.
> > > >
> > > > Thanks,
> > > > Deng
> > > >
> > > >
> > > > On Fri, Jul 11, 2008 at 6:59 PM, Maria Odea Ching <oching@apache.org
> >
> > > > wrote:
> > > >
> > > >> On Fri, Jul 11, 2008 at 6:35 PM, nicolas de loof <
> nicolas@apache.org>
> > > >> wrote:
> > > >>
> > > >>> I've looked at MetaDataTools class.It merges the metadatas from
> > > multiple
> > > >>> repositories in a target file. This could be refactored into
> another
> > > >>> method
> > > >>> to return the ArchivaRepositoryMetadata objet, so that it can be
> used
> > > for
> > > >>> repo grouping.
> > > >>
> > > >>
> > > >> I'm using the RepositoryMetadataMerge.merge(..) to merge the
> metadata
> > > from
> > > >> each repository into
> > > >> a main metadata file, so I don't need to refactor anything. I have
> > still
> > > >> yet to test it though. I'll commit it in svn asap
> > > >> so you could also take a look :)
> > > >>
> > > >> The "*write to file*" process should also be skipped when there is
> no
> > > >>> concrete metadata (availableVersions or Plugins) to avoid creating
> > > >>> "empty"
> > > >>> metadatas in managed repo for requested artifacts that aren't
> > > available.
> > > >>>
> > > >>> Could this be a security issue with archiva ? requesting a fake
> > > artifact
> > > >>> metadata creates a new File. Requesting thousand of them with a
> > random
> > > >>> name
> > > >>> will create thousand files. This could be the basis of a "brute
> > force"
> > > >>> deny
> > > >>> of service attack, with file system becoming full. Or maybe I'm a
> > > >>> paranoid ?
> > > >>
> > > >>
> > > >> This is indeed alarming! I'll try replicating this and send my
> results
> > > >> back..
> > > >>
> > > >>
> > > >>>
> > > >>> Nicolas.
> > > >>
> > > >>
> > > >> Thanks,
> > > >> Deng
> > > >>
> > > >>
> > > >>>
> > > >>> 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > > >>>
> > > >>> > Okay, thanks Nico.. I think I can do the merging stuff. Will be
> > back
> > > >>> online
> > > >>> > later when I get home..
> > > >>> >
> > > >>> > -Deng
> > > >>> >
> > > >>> > On Thu, Jul 10, 2008 at 6:14 PM, nicolas de loof <
> > nicolas@apache.org
> > > >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > 2nd option may be simplier but not really nice, let's consider
> > this
> > > >>> use
> > > >>> > > case
> > > >>> > > :
> > > >>> > >
> > > >>> > > A user has an issue with a maven plugin. As a nice guy, he
> > creates
> > > an
> > > >>> > > issue,
> > > >>> > > attach a patch, but to get quick fix he deploys a snapshot to
> its
> > > >>> > corporate
> > > >>> > > repo.
> > > >>> > >
> > > >>> > > To get it using a group repository ("public" proxying public
> > repos
> > > +
> > > >>> > > "corporate"), we need to merge the metadatas from both
> > > repositories,
> > > >>> so
> > > >>> > > that
> > > >>> > > the custom snapshot will get listed in metadatas + all
> available
> > > >>> version
> > > >>> > on
> > > >>> > > public snapshot repositories.
> > > >>> > >
> > > >>> > > From my understanding, this will require some "*list available
> > > >>> files*"
> > > >>> > > logic
> > > >>> > > + "*merge metadatas*" in ArchivaDavResource.createResource(),
> and
> > > not
> > > >>> > just
> > > >>> > > "return resource;"
> > > >>> > >
> > > >>> > > I've commited the "*list available files*" part that was
> trivial.
> > > >>> > >
> > > >>> > > I'm not used with Dav (jackRabit) API and metadata file format,
> > so
> > > >>> any
> > > >>> > help
> > > >>> > > for the "*merge metadatas*"  would be welcome (TODO at line
> 263).
> > > >>> This
> > > >>> > > would
> > > >>> > > require to create an "in memory" virtual resource for the
> merged
> > > >>> > metadata.
> > > >>> > >
> > > >>> > > Nico.
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > > >>> > >
> > > >>> > > > Oh ok, sorry I thought it was a blank file returned. I
> haven't
> > > >>> tried
> > > >>> > > > replicating it yet, was just looking through the codes.
> > > >>> > > > I guess either of the 2 options would work.. I'm still
> > finishing
> > > up
> > > >>> > some
> > > >>> > > > things in the search, I'll see what I can do for 872 later
> > > tonight.
> > > >>> > > > Unless of course you want to work on it? :)
> > > >>> > > >
> > > >>> > > > Thanks,
> > > >>> > > > Deng
> > > >>> > > >
> > > >>> > > > On Thu, Jul 10, 2008 at 5:13 PM, nicolas de loof <
> > > >>> nicolas@apache.org>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > This mean the First metadata file found in on repo of the
> > group
> > > >>> is
> > > >>> > > > returned
> > > >>> > > > > !
> > > >>> > > > >
> > > >>> > > > > This is what happen : I get a metadat file with no
> <version>
> > > >>> included
> > > >>> > > > (not
> > > >>> > > > > an blank file, but an XML construct with just root element)
> :
> > > >>> this is
> > > >>> > > the
> > > >>> > > > > content of my first repo in the group, that did not
> retrieve
> > > the
> > > >>> > > expected
> > > >>> > > > > artifact.
> > > >>> > > > >
> > > >>> > > > > I made a simple test : place "release" repo before
> > "corporate"
> > > in
> > > >>> the
> > > >>> > > > > group,
> > > >>> > > > > and I the get the expected metadata.
> > > >>> > > > >
> > > >>> > > > > My conclusion is we have two options :
> > > >>> > > > >
> > > >>> > > > > - support metadata merge from repositories in the group
> (not
> > > just
> > > >>> > > "first
> > > >>> > > > > win")
> > > >>> > > > > - don't create metadata in managed repo if no artifact was
> > > found
> > > >>> from
> > > >>> > > > > proxies.
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > >
> > > >>> > > > > 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > > >>> > > > >
> > > >>> > > > > > The ArchivaVirtualDavResource is for the dav browse
> > > >>> (directories),
> > > >>> > > not
> > > >>> > > > > for
> > > >>> > > > > > artifact requests..
> > > >>> > > > > >
> > > >>> > > > > > Hmm, the virtual repos work this way:
> > > >>> > > > > > - client requests for an artifact
> > > >>> > > > > > - RepoServlet handles the request and goes to
> > > >>> > > ArchivaDavSessionProvider
> > > >>> > > > > > - in ArchivaDavSessionProvider, the request is checked if
> > it
> > > is
> > > >>> a
> > > >>> > > repo
> > > >>> > > > > > group
> > > >>> > > > > > url or a regular repo url.
> > > >>> > > > > >   If it is, then loop through the repos under the group.
> > The
> > > >>> first
> > > >>> > > > > > requested artifact found among the repos
> > > >>> > > > > >   is returned (proxied or already existing). The metadata
> > > >>> > > update/merge
> > > >>> > > > > > happens when the artifact is
> > > >>> > > > > >   fetched from the proxies (but this is on a per
> repository
> > > >>> level,
> > > >>> > > > > there's
> > > >>> > > > > > no metadata of metadata files
> > > >>> > > > > >   from each repo being merged at the repo group level)
> > > >>> > > > > >
> > > >>> > > > > > The same process goes if the request made is for a
> metadata
> > > >>> file. I
> > > >>> > > > don't
> > > >>> > > > > > know why a blank metadata file was returned.. :(
> > > >>> > > > > >
> > > >>> > > > > > Thanks,
> > > >>> > > > > > Deng
> > > >>> > > > > >
> > > >>> > > > > > On Thu, Jul 10, 2008 at 3:39 PM, nicolas de loof <
> > > >>> > nicolas@apache.org
> > > >>> > > >
> > > >>> > > > > > wrote:
> > > >>> > > > > >
> > > >>> > > > > > > I just started investigating on this. I can't find
> where
> > > the
> > > >>> > > > metadatas
> > > >>> > > > > > are
> > > >>> > > > > > > merged when a repository group is set. From my
> > > understanding,
> > > >>> > > > > > > ArchivaVirtualDavResource is used with a List<File> of
> > > >>> metadatas
> > > >>> > > from
> > > >>> > > > > > each
> > > >>> > > > > > > repo in the group. But I don't find WHERE the metadata
> > > files
> > > >>> are
> > > >>> > > > merged
> > > >>> > > > > > > into
> > > >>> > > > > > > a single XML content...
> > > >>> > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > > > 2008/7/10 Maria Odea Ching <oching@exist.com>:
> > > >>> > > > > > >
> > > >>> > > > > > > > Ok, thanks Nico, Dan :)
> > > >>> > > > > > > > I'll look into this further.. I guess we could
> include
> > > this
> > > >>> in
> > > >>> > > > 1.1.1
> > > >>> > > > > > > which
> > > >>> > > > > > > > is scheduled to be released right after 1.1 to fix
> > > another
> > > >>> > > blocker
> > > >>> > > > in
> > > >>> > > > > > > 1.1.
> > > >>> > > > > > > >
> > > >>> > > > > > > > Thanks,
> > > >>> > > > > > > > Deng
> > > >>> > > > > > > >
> > > >>> > > > > > > > On Thu, Jul 10, 2008 at 2:37 PM, nicolas de loof <
> > > >>> > > > nicolas@apache.org
> > > >>> > > > > >
> > > >>> > > > > > > > wrote:
> > > >>> > > > > > > >
> > > >>> > > > > > > > > MRM-872 created for this, with the concrete case I
> > fall
> > > >>> into.
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > If confirmed, this is a blocker for repository
> group
> > > use
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > 2008/7/10 Dan Tran <dantran@gmail.com>:
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > > sounds like  a blocking bug? would love to see
> this
> > > >>> fixed
> > > >>> > in
> > > >>> > > > 1.1
> > > >>> > > > > > :-)
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > On Wed, Jul 9, 2008 at 7:11 PM, Maria Odea Ching
> <
> > > >>> > > > > > oching@apache.org>
> > > >>> > > > > > > > > > wrote:
> > > >>> > > > > > > > > > > Hi Nico,
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > That looks like a new bug :( The metadata
> should
> > be
> > > >>> just
> > > >>> > > the
> > > >>> > > > > same
> > > >>> > > > > > > as
> > > >>> > > > > > > > > the
> > > >>> > > > > > > > > > > regular metadata of a managed repo (not
> belonging
> > > to
> > > >>> a
> > > >>> > > group)
> > > >>> > > > > as
> > > >>> > > > > > it
> > > >>> > > > > > > > > uses
> > > >>> > > > > > > > > > the
> > > >>> > > > > > > > > > > same code for webdav & proxying. Could you file
> a
> > > >>> jira
> > > >>> > for
> > > >>> > > > > this?
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > Thanks,
> > > >>> > > > > > > > > > > Deng
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > On Wed, Jul 9, 2008 at 8:45 PM, nicolas de loof
> <
> > > >>> > > > > > > nicolas@apache.org>
> > > >>> > > > > > > > > > wrote:
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >> Hy,
> > > >>> > > > > > > > > > >>
> > > >>> > > > > > > > > > >> I get strange behaviour wen using repository
> > group
> > > :
> > > >>> > > > > > > > > > >>
> > > >>> > > > > > > > > > >> From a fresh maven installation (empty local
> > > repo),
> > > >>> with
> > > >>> > > > > > settings
> > > >>> > > > > > > > set
> > > >>> > > > > > > > > to
> > > >>> > > > > > > > > > >> mirror central to my archiva repository group
> > URL,
> > > >>> the
> > > >>> > > > eclipse
> > > >>> > > > > > 2.5
> > > >>> > > > > > > > > > plugin
> > > >>> > > > > > > > > > >> can't find the required
> > > >>> > > > > org.eclipse.core:resources:[3.1.0,4.0.0)
> > > >>> > > > > > > > > > >>
> > > >>> > > > > > > > > > >> The maven-metadata.xml returned by the group
> URL
> > > is
> > > >>> > emty.
> > > >>> > > > > > > requesting
> > > >>> > > > > > > > > > >> metadatas from individual managed repos in the
> > > group
> > > >>> are
> > > >>> > > > fine.
> > > >>> > > > > > Is
> > > >>> > > > > > > > this
> > > >>> > > > > > > > > a
> > > >>> > > > > > > > > > >> known limitation ?
> > > >>> > > > > > > > > > >>
> > > >>> > > > > > > > > > >> Nico.
> > > >>> > > > > > > > > > >>
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

------=_Part_11164_7815718.1216198566184--