From ivy-user-return-7879-apmail-ant-ivy-user-archive=ant.apache.org@ant.apache.org Mon May 2 21:07:22 2011 Return-Path: X-Original-To: apmail-ant-ivy-user-archive@www.apache.org Delivered-To: apmail-ant-ivy-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A4D1B3CD5 for ; Mon, 2 May 2011 21:07:22 +0000 (UTC) Received: (qmail 960 invoked by uid 500); 2 May 2011 21:07:22 -0000 Delivered-To: apmail-ant-ivy-user-archive@ant.apache.org Received: (qmail 933 invoked by uid 500); 2 May 2011 21:07:22 -0000 Mailing-List: contact ivy-user-help@ant.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: ivy-user@ant.apache.org Delivered-To: mailing list ivy-user@ant.apache.org Delivered-To: moderator for ivy-user@ant.apache.org Received: (qmail 79483 invoked by uid 99); 2 May 2011 20:55:54 -0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) X-VirusChecked: Checked X-Env-Sender: jose.nunez-zuleta@barclayscapital.com X-Msg-Ref: server-15.tower-139.messagelabs.com!1304369725!9498860!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [146.127.253.25] From: To: , Date: Mon, 2 May 2011 16:55:20 -0400 Subject: RE: How to create a filesystem repository Thread-Topic: How to create a filesystem repository Thread-Index: AcwJCWP7vUo07QfmRhCNhXa7WI9kMAAASomg Message-ID: <34922D8098CB7048A99568D009AF262D08371C8EB2@NYKPCMMGMB05.INTRANET.BARCAPINT.COM> References: <34922D8098CB7048A99568D009AF262D08371C8EAC@NYKPCMMGMB05.INTRANET.BARCAPINT.COM> <4DBF1713.4020204@masergy.com> In-Reply-To: <4DBF1713.4020204@masergy.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hello Kirby, Bootstrapping to an existing repository is the best way? By bootstrapping= =20a monstrous repository will be created with a lot of stuff that is not= =20really needed in the end. How difficult is to create an 'empty reposit= ory' and fill it with the right contents? (your Perl script is definitely= =20useful, same as the ant task to publish to the new repository). I'm trying to follow this short tutorial: http://www.drillio.com/en/software-build/apache-ivy/create-local-reposito= ry-guide/ Not exactly what I need but I like the fact that you are in control on wh= at is inside the repository. I'll let you know how far I get with this :-= ) Anybody else has other suggestions? Thanks in advance! --Jose -----Original Message----- From: Kirby Files [mailto:kfiles@masergy.com]=20 Sent: 02 May 2011 16:42 To: ivy-user@ant.apache.org Cc: Nunez Zuleta, Jose V: IT (NYK) Subject: Re: How to create a filesystem repository jose.nunez-zuleta@barclayscapital.com wrote on 05/02/2011 03:00 PM: > Hello to all, > > Can someone tell me how I can create a local filesystem repository with= =20Ivy? I've been through the documentation and it is not clear how I can= =20get started. > > My jar files are in the format > > Libdirectory/module/jar1.jar > Libdirectory/module/jar2.jar Yes, Jose, this is one of the more frustrating deficiencies in the Ivy=20 toolset: it's difficult to migrate from the most common situation (a=20 directory full of jars with no metadata) to an ivy.xml file with=20 dependencies and a repository containing those jars with metadata. I have some suggestions: 1) Start with some decent metadata and existing sources of dependency=20 management: use the IvyRoundup packager repository or the public Maven=20 repos (with the ibiblio resolver) to bootstrap your filesystem=20 repository with the task (it can copy from a source repo=20 to a dest repo, if necessary converting the maven poms to ivy.xml in=20 the process). 2) Create an ivysettings.xml with all of the repos, public and=20 private, that you'll need to access. 3) Create an ivy.xml and a ivy:publish target for each of your=20 projects, and publish those to your repository. 3) Use scripts to install jars not available publicly, nor produced=20 internally, from a directory to your repository. I use this script: -------------------------------------- #!/bin/csh if ($1 =3D=3D "") then =20 echo "Usage: installjar " else =20 /usr/local/ant/bin/ant -Dorg=3D"$1" -Dmodule=3D"$2" -Drev=3D"$3"=20 -Dinstall.path=3D"$4" $5 $6 $7 $8 $9 endif -------------------------------------- With the following build target: -------------------------------------- =20 install jar to masrep"> =20 =20 -------------------------------------- Where my build.properties defines these variables: localjar.resolver=3Dlocaljars to.resolver=3Dmasrep_sftp And my ivysettings.xml includes: =20 =20 =20 =20 =20 =20 This script allows me to specify a jar on the local filesystem by its=20 name and the directory in which it resides (install.path), add the=20 org, and rev metadata, and publish it to one of my resolvers (I use=20 SFTP with a keyfile to push files up to our webserver; you may use a=20 filesystem resolver or webdav as a destination). I also have a bigger hack of a perl script which attempts to read a=20 directory of jars and guess the org, modul, and rev (validating=20 against maven2), or prompt the user when it's unclear. See below. Thanks, --- Kirby Files Software Architect Masergy Communications kfiles@masergy.com ---------------------------------------------- #!/usr/bin/perl # Looks for jars in the incoming directory, attemps to parse the # needed ivy info from the filename, and if successful, publishes # the jar to the Masergy ivrep repository. # If using linux with package mgmt, dependencies are: # perl-libwww-perl # perl-XML-Xpath # perl-Archive-Zip use LWP::UserAgent; use XML::XPath; use XML::XPath::XMLParser; use Archive::Zip; use Cwd; use strict qw(vars); use vars qw($indir $ant $rep $pre $post); $indir =3D shift(@ARGV); if ($indir !~ m#^/#) { =20 my $cwd =3D getcwd(); =20 $indir =3D~ s/^/$cwd\//; =20 print "CWD: $indir\n"; } $ant =3D "/usr/local/ant/bin/ant"; $rep =3D "newjars"; $pre=3D""; open (DIR, "ls -1 $indir|") || die "could not open $indir\n"; while () { =20 my ($org, $module, $jar, $ver); =20 next unless /jar/; =20 chomp; =20 my $file =3D $indir . "/" . $_; =20 $org=3D$_; =20 $module=3D$_; =20 $jar=3D$_; =20 $ver=3D$_; =20 $module=3D~s/^(.*)\.jar/$1/; =20 $module=3D~s/([^.]*)-\d.*/$1/; =20 $ver=3D~s/^[^.]*-([\d.]+.*)\.jar/$1/; =20 $ver=3D"unknown" unless $ver =3D~ /\d/; =20 ## Figure out the org =20 ## Ask the user, if there's ambiguity =20 $org =3D &maven_search_org( $module ); =20 if (-1 =3D=3D $org) { =20 $org =3D &guess_org($module); =20 print "Org guessed from module name:\n $org\n"; =20 $org =3D &classes_org($file); =20 print "Org guessed from jar package naming:\n $org\n" =20 unless (-1 =3D=3D $org); =20 do { =20 $org =3D &get_user_org(); =20 } while ($org =3D~ /^$/); =20 } =20 if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) { =20 print "Skipping jar $jar:\n"; =20 print " $org / $module / $ver\n"; =20 next; =20 } else { =20 &publish_jar($jar, $org, $module, $ver); =20 } =20 #&write_ivy_conf($jar, $org, $module, $ver); } ## Use ivy::install to publish the jar via sftp to ivyrep sub publish_jar { =20 my ($jar, $org, $module, $ver) =3D @_; =20 print "Uploading $org / $module / $jar ... "; =20 my $resp =3D `$ant -Dorg=3D$org -Dmodule=3D$module -Drev=3D$ver=20 -Dinstall.path=3D$indir 2>&1`; =20 if ($resp =3D~ /failed/i) { =20 print STDERR "Error uploading $jar:\n", $resp; =20 print "\n"; =20 } else { =20 print "Done.\n"; =20 } } ## Find a directory in the jar to propose as the org ## searches for any dirs with two path elements sub classes_org { =20 my ($file) =3D @_; =20 my $somezip =3D Archive::Zip->new(); =20 unless ( $somezip->read( $file ) =3D=3D AZ_OK ) { =20 die "Error reading jarfile $file"; =20 } =20 my @members =3D $somezip->members(); =20 my @dirnames =3D (); =20 foreach (@members) { =20 if ($_->isDirectory()) { =20 my $fname =3D $_->fileName(); =20 if ($fname =3D~ /^\w+\/\w+\/$/) { =20 $fname =3D~ s/^(\w+)\/(\w+)\/$/$1.$2/; =20 return $fname; =20 } =20 } =20 } =20 return -1; } ## Ask the user for the best org name sub get_user_org { =20 print "Enter preffered org:> "; =20 my $line =3D ; =20 chomp($line); =20 return $line; } ## Make a guess about the org name from the module name sub guess_org { =20 my ($module) =3D @_; =20 my $org =3D $module; =20 $org=3D~s/spring.*/springframework/; =20 $org=3D~s/hiber.*/hibernate/; =20 $org=3D~s/standard.*/taglibs/; =20 $org=3D~s/jstl.*/taglibs/; =20 $org=3D~s/persistence.*/sun/; =20 $org=3D~s/jta.*/sun/; =20 $org=3D~s/servlet-api.*/tomcat/; =20 $org=3D~s/ojdbc.*/oracle/; =20 $org=3D~s/cglib.*/cglib/; =20 $org=3D~s/javassist.*/jboss/; =20 $org=3D~s/([^.]*)-\d.*jar/$1/; =20 $org=3D~s/(.*)\.jar/$1/; =20 return $org; } ## Search the maven repo for a module matching this jar ## If found, return the module's organization ## if none, or more than one, are found, ask the user sub maven_search_org { =20 my ($module) =3D @_; =20 my $ua =3D LWP::UserAgent->new; =20 $ua->agent('Maven Repo Search'); =20 my $service=3D=20 'http://maven.ozacc.com/search?format=3Dxml&type=3Djar&keyword=3D'; =20 my $url=3DURI->new($service . $module); =20 my $req=3DHTTP::Request->new; =20 $req->method('GET'); =20 $req->uri($url); =20 my $resp =3D $ua->request($req)->content; =20 my $org =3D &parse_resp( $module, $resp ); =20 return $org; } ## Parse the XML response from the Maven search service sub parse_resp { =20 my ($module, $data) =3D @_; =20 my $xp =3D XML::XPath->new(xml =3D> $data); =20 my $nodeset =3D $xp->find('//groupId'); # find all paragraphs =20 # Get rid of duplicates =20 my %uniq=3D(); =20 foreach my $node ($nodeset->get_nodelist) { =20 $uniq{ $xp->getNodeText( $node ) } =3D 1; =20 } =20 my @keys =3D (sort keys %uniq); =20 if ($#keys =3D=3D 0) { =20 return ($keys[0]); =20 } elsif ($#keys =3D=3D -1) { =20 print "Found no module matching $module in Maven repo\n"; =20 return -1; =20 } else { =20 print "Found multiple possible orgs for $module in Maven repo:\n";= =20 foreach (@keys) { =20 print " $_\n"; =20 } =20 return -1; =20 } } ## (obsolete) Write ant task and ivy.xml conf lines to create repo sub write_ivy_conf { =20 my ($jar, $org, $module, $ver) =3D @_; =20 my $line =3D $pre . $org . "\" module=3D\"" . $module . "\"=20 revision=3D\"" . $ver . "\"". $post . "\n"; =20 system("mkdir -p $rep/$org/$module/jars"); =20 link "$indir/$jar", "$rep/$org/$module/jars/$jar"; =20 print $line; =20 print "\n"; } _______________________________________________ This e-mail may contain information that is confidential, privileged or o= therwise protected from disclosure. If you are not an intended recipient = of this e-mail, do not duplicate or redistribute it by any means. Please = delete it and any attachments and notify the sender that you have receive= d it in error. Unless specifically indicated, this e-mail is not an offer= =20to buy or sell or a solicitation to buy or sell any securities, invest= ment products or other financial product or service, an official confirma= tion of any transaction, or an official statement of Barclays. Any views = or opinions presented are solely those of the author and do not necessari= ly represent those of Barclays. This e-mail is subject to terms available= =20at the following link: www.barcap.com/emaildisclaimer. By messaging wi= th Barclays you consent to the foregoing. Barclays Capital is the invest= ment banking division of Barclays Bank PLC, a company registered in Engla= nd (number 1026167) with its registered office at 1 Churchill Place, Lond= on, E14 5HP. This email may relate to or be sent from other members of t= he Barclays Group. _______________________________________________