Mailing-List: contact dev-help@archiva.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@archiva.apache.org
Received-SPF: pass (athena.apache.org: domain of emmanuel.venisse@gmail.com
 designates 66.249.82.227 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=BW9DGSUnjmIUhuB+1T6A2pb8KF2QRe8RQXiHhXC1i4eQWyoq41kROR8D4koV+35QIh
         X1qnW1g+y0FRPCN4FDppoLJ3j8bjsgm5JFFZxAdTz1s1pjAcR0Tw+gzTUbZfkTX2pMWB
         Ktxq7WBXqv19OGL2J841I34yOpLx0yJUu3OAY=
Message-ID: <f345f3480807160147m74211ca4gbd89e5bd9a3db4ef@mail.gmail.com>
Date: Wed, 16 Jul 2008 10:47:18 +0200
From: "Emmanuel Venisse" <emmanuel.venisse@gmail.com>
To: dev@archiva.apache.org
Subject: Re: wrong metadatas
In-Reply-To: <4c39e3030807160125j1a04b1e5nc8e336ed3a7496cf@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_11061_11296454.1216198039043"
References: <4c39e3030807090545u1bbc192fw3e4189a48b0d1086@mail.gmail.com>
	 <4c39e3030807100213pf49f04cv71d52818abe24aa2@mail.gmail.com>
	 <1a57a2980807100247n72066c89jf18a8c95de5fe23@mail.gmail.com>
	 <4c39e3030807100314x447cadbdl4b6bc33d0486eef0@mail.gmail.com>
	 <1a57a2980807100344j66eae939mcc095ceab218f805@mail.gmail.com>
	 <4c39e3030807110335g7807d828y6ddea2ea6d69d44d@mail.gmail.com>
	 <1a57a2980807110359u5e9eabedhd869d00786f9529e@mail.gmail.com>
	 <1a57a2980807130708g2037722fs8918033157dea91c@mail.gmail.com>
	 <1a57a2980807140225r547b9bcfs3ef25c0be122520f@mail.gmail.com>
	 <4c39e3030807160125j1a04b1e5nc8e336ed3a7496cf@mail.gmail.com>

------=_Part_11061_11296454.1216198039043
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

If it's ok, it is possible to restart the release or do you want to add more
things in 1.1?

Emmanuel

On Wed, Jul 16, 2008 at 10:25 AM, nicolas de loof <nicolas@apache.org>
wrote:

> I made some tests and this works nicely for me.
>
> Thanks a lot !
>
> 2008/7/14 Maria Odea Ching <oching@apache.org>:
>
> > Additional fixes in -r676516 and -r676526..
> > Summary of the fixes:
> >
> >   1. generate checksums for the generated merged metadata
> >   2. put the merged metadata & its checksums in the first repo where the
> >   metadata was found. the filename for the merged metadata is
> >   maven-metadata-${repoGroupId}.xml to handle instances when a repository
> >   belongs to multiple groups.
> >   3. merge the metadata if and only if all the following conditions are
> >   met:
> >      - the request was made to a repository group url
> >      - the requested resource is a metadata file or a checksum file of a
> >      metadata
> >      - the requested metadata file is at the project level, not version
> >      level
> >   4. added test case when metadata checksums are requested after the
> >   metadata file (simulate Maven requests)
> >
> > Thanks,
> > Deng
> >
> > On Sun, Jul 13, 2008 at 10:08 PM, Maria Odea Ching <oching@apache.org>
> > wrote:
> >
> > > Hi Nico,
> > >
> > > Could you verify the fix i committed in trunk -r676322 if it works for
> > you?
> > > I tested it out  using ranges and I was able to get the artifacts from
> > the
> > > different repos.
> > >
> > > Thanks,
> > > Deng
> > >
> > >
> > > On Fri, Jul 11, 2008 at 6:59 PM, Maria Odea Ching <oching@apache.org>
> > > wrote:
> > >
> > >> On Fri, Jul 11, 2008 at 6:35 PM, nicolas de loof <nicolas@apache.org>
> > >> wrote:
> > >>
> > >>> I've looked at MetaDataTools class.It merges the metadatas from
> > multiple
> > >>> repositories in a target file. This could be refactored into another
> > >>> method
> > >>> to return the ArchivaRepositoryMetadata objet, so that it can be used
> > for
> > >>> repo grouping.
> > >>
> > >>
> > >> I'm using the RepositoryMetadataMerge.merge(..) to merge the metadata
> > from
> > >> each repository into
> > >> a main metadata file, so I don't need to refactor anything. I have
> still
> > >> yet to test it though. I'll commit it in svn asap
> > >> so you could also take a look :)
> > >>
> > >> The "*write to file*" process should also be skipped when there is no
> > >>> concrete metadata (availableVersions or Plugins) to avoid creating
> > >>> "empty"
> > >>> metadatas in managed repo for requested artifacts that aren't
> > available.
> > >>>
> > >>> Could this be a security issue with archiva ? requesting a fake
> > artifact
> > >>> metadata creates a new File. Requesting thousand of them with a
> random
> > >>> name
> > >>> will create thousand files. This could be the basis of a "brute
> force"
> > >>> deny
> > >>> of service attack, with file system becoming full. Or maybe I'm a
> > >>> paranoid ?
> > >>
> > >>
> > >> This is indeed alarming! I'll try replicating this and send my results
> > >> back..
> > >>
> > >>
> > >>>
> > >>> Nicolas.
> > >>
> > >>
> > >> Thanks,
> > >> Deng
> > >>
> > >>
> > >>>
> > >>> 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > >>>
> > >>> > Okay, thanks Nico.. I think I can do the merging stuff. Will be
> back
> > >>> online
> > >>> > later when I get home..
> > >>> >
> > >>> > -Deng
> > >>> >
> > >>> > On Thu, Jul 10, 2008 at 6:14 PM, nicolas de loof <
> nicolas@apache.org
> > >
> > >>> > wrote:
> > >>> >
> > >>> > > 2nd option may be simplier but not really nice, let's consider
> this
> > >>> use
> > >>> > > case
> > >>> > > :
> > >>> > >
> > >>> > > A user has an issue with a maven plugin. As a nice guy, he
> creates
> > an
> > >>> > > issue,
> > >>> > > attach a patch, but to get quick fix he deploys a snapshot to its
> > >>> > corporate
> > >>> > > repo.
> > >>> > >
> > >>> > > To get it using a group repository ("public" proxying public
> repos
> > +
> > >>> > > "corporate"), we need to merge the metadatas from both
> > repositories,
> > >>> so
> > >>> > > that
> > >>> > > the custom snapshot will get listed in metadatas + all available
> > >>> version
> > >>> > on
> > >>> > > public snapshot repositories.
> > >>> > >
> > >>> > > From my understanding, this will require some "*list available
> > >>> files*"
> > >>> > > logic
> > >>> > > + "*merge metadatas*" in ArchivaDavResource.createResource(), and
> > not
> > >>> > just
> > >>> > > "return resource;"
> > >>> > >
> > >>> > > I've commited the "*list available files*" part that was trivial.
> > >>> > >
> > >>> > > I'm not used with Dav (jackRabit) API and metadata file format,
> so
> > >>> any
> > >>> > help
> > >>> > > for the "*merge metadatas*"  would be welcome (TODO at line 263).
> > >>> This
> > >>> > > would
> > >>> > > require to create an "in memory" virtual resource for the merged
> > >>> > metadata.
> > >>> > >
> > >>> > > Nico.
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > >>> > >
> > >>> > > > Oh ok, sorry I thought it was a blank file returned. I haven't
> > >>> tried
> > >>> > > > replicating it yet, was just looking through the codes.
> > >>> > > > I guess either of the 2 options would work.. I'm still
> finishing
> > up
> > >>> > some
> > >>> > > > things in the search, I'll see what I can do for 872 later
> > tonight.
> > >>> > > > Unless of course you want to work on it? :)
> > >>> > > >
> > >>> > > > Thanks,
> > >>> > > > Deng
> > >>> > > >
> > >>> > > > On Thu, Jul 10, 2008 at 5:13 PM, nicolas de loof <
> > >>> nicolas@apache.org>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > This mean the First metadata file found in on repo of the
> group
> > >>> is
> > >>> > > > returned
> > >>> > > > > !
> > >>> > > > >
> > >>> > > > > This is what happen : I get a metadat file with no <version>
> > >>> included
> > >>> > > > (not
> > >>> > > > > an blank file, but an XML construct with just root element) :
> > >>> this is
> > >>> > > the
> > >>> > > > > content of my first repo in the group, that did not retrieve
> > the
> > >>> > > expected
> > >>> > > > > artifact.
> > >>> > > > >
> > >>> > > > > I made a simple test : place "release" repo before
> "corporate"
> > in
> > >>> the
> > >>> > > > > group,
> > >>> > > > > and I the get the expected metadata.
> > >>> > > > >
> > >>> > > > > My conclusion is we have two options :
> > >>> > > > >
> > >>> > > > > - support metadata merge from repositories in the group (not
> > just
> > >>> > > "first
> > >>> > > > > win")
> > >>> > > > > - don't create metadata in managed repo if no artifact was
> > found
> > >>> from
> > >>> > > > > proxies.
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > 2008/7/10 Maria Odea Ching <oching@apache.org>:
> > >>> > > > >
> > >>> > > > > > The ArchivaVirtualDavResource is for the dav browse
> > >>> (directories),
> > >>> > > not
> > >>> > > > > for
> > >>> > > > > > artifact requests..
> > >>> > > > > >
> > >>> > > > > > Hmm, the virtual repos work this way:
> > >>> > > > > > - client requests for an artifact
> > >>> > > > > > - RepoServlet handles the request and goes to
> > >>> > > ArchivaDavSessionProvider
> > >>> > > > > > - in ArchivaDavSessionProvider, the request is checked if
> it
> > is
> > >>> a
> > >>> > > repo
> > >>> > > > > > group
> > >>> > > > > > url or a regular repo url.
> > >>> > > > > >   If it is, then loop through the repos under the group.
> The
> > >>> first
> > >>> > > > > > requested artifact found among the repos
> > >>> > > > > >   is returned (proxied or already existing). The metadata
> > >>> > > update/merge
> > >>> > > > > > happens when the artifact is
> > >>> > > > > >   fetched from the proxies (but this is on a per repository
> > >>> level,
> > >>> > > > > there's
> > >>> > > > > > no metadata of metadata files
> > >>> > > > > >   from each repo being merged at the repo group level)
> > >>> > > > > >
> > >>> > > > > > The same process goes if the request made is for a metadata
> > >>> file. I
> > >>> > > > don't
> > >>> > > > > > know why a blank metadata file was returned.. :(
> > >>> > > > > >
> > >>> > > > > > Thanks,
> > >>> > > > > > Deng
> > >>> > > > > >
> > >>> > > > > > On Thu, Jul 10, 2008 at 3:39 PM, nicolas de loof <
> > >>> > nicolas@apache.org
> > >>> > > >
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > I just started investigating on this. I can't find where
> > the
> > >>> > > > metadatas
> > >>> > > > > > are
> > >>> > > > > > > merged when a repository group is set. From my
> > understanding,
> > >>> > > > > > > ArchivaVirtualDavResource is used with a List<File> of
> > >>> metadatas
> > >>> > > from
> > >>> > > > > > each
> > >>> > > > > > > repo in the group. But I don't find WHERE the metadata
> > files
> > >>> are
> > >>> > > > merged
> > >>> > > > > > > into
> > >>> > > > > > > a single XML content...
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > 2008/7/10 Maria Odea Ching <oching@exist.com>:
> > >>> > > > > > >
> > >>> > > > > > > > Ok, thanks Nico, Dan :)
> > >>> > > > > > > > I'll look into this further.. I guess we could include
> > this
> > >>> in
> > >>> > > > 1.1.1
> > >>> > > > > > > which
> > >>> > > > > > > > is scheduled to be released right after 1.1 to fix
> > another
> > >>> > > blocker
> > >>> > > > in
> > >>> > > > > > > 1.1.
> > >>> > > > > > > >
> > >>> > > > > > > > Thanks,
> > >>> > > > > > > > Deng
> > >>> > > > > > > >
> > >>> > > > > > > > On Thu, Jul 10, 2008 at 2:37 PM, nicolas de loof <
> > >>> > > > nicolas@apache.org
> > >>> > > > > >
> > >>> > > > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > MRM-872 created for this, with the concrete case I
> fall
> > >>> into.
> > >>> > > > > > > > >
> > >>> > > > > > > > > If confirmed, this is a blocker for repository group
> > use
> > >>> > > > > > > > >
> > >>> > > > > > > > > 2008/7/10 Dan Tran <dantran@gmail.com>:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > sounds like  a blocking bug? would love to see this
> > >>> fixed
> > >>> > in
> > >>> > > > 1.1
> > >>> > > > > > :-)
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Wed, Jul 9, 2008 at 7:11 PM, Maria Odea Ching <
> > >>> > > > > > oching@apache.org>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > > > Hi Nico,
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > That looks like a new bug :( The metadata should
> be
> > >>> just
> > >>> > > the
> > >>> > > > > same
> > >>> > > > > > > as
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > regular metadata of a managed repo (not belonging
> > to
> > >>> a
> > >>> > > group)
> > >>> > > > > as
> > >>> > > > > > it
> > >>> > > > > > > > > uses
> > >>> > > > > > > > > > the
> > >>> > > > > > > > > > > same code for webdav & proxying. Could you file a
> > >>> jira
> > >>> > for
> > >>> > > > > this?
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Thanks,
> > >>> > > > > > > > > > > Deng
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > On Wed, Jul 9, 2008 at 8:45 PM, nicolas de loof <
> > >>> > > > > > > nicolas@apache.org>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >> Hy,
> > >>> > > > > > > > > > >>
> > >>> > > > > > > > > > >> I get strange behaviour wen using repository
> group
> > :
> > >>> > > > > > > > > > >>
> > >>> > > > > > > > > > >> From a fresh maven installation (empty local
> > repo),
> > >>> with
> > >>> > > > > > settings
> > >>> > > > > > > > set
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > >> mirror central to my archiva repository group
> URL,
> > >>> the
> > >>> > > > eclipse
> > >>> > > > > > 2.5
> > >>> > > > > > > > > > plugin
> > >>> > > > > > > > > > >> can't find the required
> > >>> > > > > org.eclipse.core:resources:[3.1.0,4.0.0)
> > >>> > > > > > > > > > >>
> > >>> > > > > > > > > > >> The maven-metadata.xml returned by the group URL
> > is
> > >>> > emty.
> > >>> > > > > > > requesting
> > >>> > > > > > > > > > >> metadatas from individual managed repos in the
> > group
> > >>> are
> > >>> > > > fine.
> > >>> > > > > > Is
> > >>> > > > > > > > this
> > >>> > > > > > > > > a
> > >>> > > > > > > > > > >> known limitation ?
> > >>> > > > > > > > > > >>
> > >>> > > > > > > > > > >> Nico.
> > >>> > > > > > > > > > >>
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

------=_Part_11061_11296454.1216198039043--