Return-Path: 
 <ivy-user-return-7877-apmail-ant-ivy-user-archive=ant.apache.org@ant.apache.org>
X-Original-To: apmail-ant-ivy-user-archive@www.apache.org
Delivered-To: apmail-ant-ivy-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 657033E2B
	for <apmail-ant-ivy-user-archive@www.apache.org>;
 Mon,  2 May 2011 20:42:24 +0000 (UTC)
Received: (qmail 62780 invoked by uid 500); 2 May 2011 20:42:24 -0000
Delivered-To: apmail-ant-ivy-user-archive@ant.apache.org
Received: (qmail 62741 invoked by uid 500); 2 May 2011 20:42:24 -0000
Mailing-List: contact ivy-user-help@ant.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:ivy-user-help@ant.apache.org>
List-Unsubscribe: <mailto:ivy-user-unsubscribe@ant.apache.org>
List-Post: <mailto:ivy-user@ant.apache.org>
List-Id: <ivy-user.ant.apache.org>
Reply-To: ivy-user@ant.apache.org
Delivered-To: mailing list ivy-user@ant.apache.org
Received: (qmail 62733 invoked by uid 99); 2 May 2011 20:42:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:42:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Kfiles@masergy.com designates
 64.47.5.26 as permitted sender)
Received: from [64.47.5.26] (HELO SMTP2.MASERGY.COM) (64.47.5.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:42:18 +0000
Received: from MTXHUB2.add0.masergy.com (10.20.1.170) by SMTP2.MASERGY.COM
 (64.47.5.26) with Microsoft SMTP Server (TLS) id 8.3.137.0; Mon, 2 May 2011
 15:41:53 -0500
Received: from [64.47.104.72] (64.47.104.72) by smtp.masergy.com (10.20.1.175)
 with Microsoft SMTP Server id 8.3.137.0; Mon, 2 May 2011 15:41:57 -0500
Message-ID: <4DBF1713.4020204@masergy.com>
Date: Mon, 2 May 2011 16:41:55 -0400
From: Kirby Files <kfiles@masergy.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: "ivy-user@ant.apache.org" <ivy-user@ant.apache.org>
CC: "jose.nunez-zuleta@barclayscapital.com"
 <jose.nunez-zuleta@barclayscapital.com>
Subject: Re: How to create a filesystem repository
References: 
 <34922D8098CB7048A99568D009AF262D08371C8EAC@NYKPCMMGMB05.INTRANET.BARCAPINT.COM>
In-Reply-To: 
 <34922D8098CB7048A99568D009AF262D08371C8EAC@NYKPCMMGMB05.INTRANET.BARCAPINT.COM>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit

jose.nunez-zuleta@barclayscapital.com wrote on 05/02/2011 03:00 PM:
> Hello to all,
>
> Can someone tell me how I can create a local filesystem repository with Ivy? I've been through the documentation and it is not clear how I can get started.
>
> My jar files are in the format
>
> Libdirectory/module/jar1.jar
> Libdirectory/module/jar2.jar

Yes, Jose, this is one of the more frustrating deficiencies in the Ivy 
toolset: it's difficult to migrate from the most common situation (a 
directory full of jars with no metadata) to an ivy.xml file with 
dependencies and a repository containing those jars with metadata.

I have some suggestions:

1) Start with some decent metadata and existing sources of dependency 
management: use the IvyRoundup packager repository or the public Maven 
repos (with the ibiblio resolver) to bootstrap your filesystem 
repository with the <ivy:install> task (it can copy from a source repo 
to a dest repo, if necessary converting the maven poms to ivy.xml in 
the process).

2) Create an ivysettings.xml with all of the repos, public and 
private, that you'll need to access.

3) Create an ivy.xml and a ivy:publish target for each of your 
projects, and publish those to your repository.

3) Use scripts to install jars not available publicly, nor produced 
internally, from a directory to your repository.

I use this script:

--------------------------------------
#!/bin/csh

if ($1 == "") then
     echo "Usage: installjar <org> <module> <revision> <path-to-file>"
else
     /usr/local/ant/bin/ant -Dorg="$1" -Dmodule="$2" -Drev="$3" 
-Dinstall.path="$4" $5 $6 $7 $8 $9
endif
--------------------------------------

With the following build target:

--------------------------------------
   <target name="installjar"
           depends="init-ivy"
           description="--> install jar to masrep">
     <ivy:install settingsRef="masergy.settings" organisation="${org}" 
module="${module}" revision="${rev}" type="jar" 
from="${localjar.resolver}" to="${to.resolver}" />
   </target>
--------------------------------------

Where my build.properties defines these variables:

localjar.resolver=localjars
to.resolver=masrep_sftp

And my ivysettings.xml includes:
   <resolvers>
     <filesystem name="localjars">
       <artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
       <artifact pattern="${install.path}/[artifact].[ext]" />
     </filesystem>
   </resolvers>

This script allows me to specify a jar on the local filesystem by its 
name and the directory in which it resides (install.path), add the 
org, and rev metadata, and publish it to one of my resolvers (I use 
SFTP with a keyfile to push files up to our webserver; you may use a 
filesystem resolver or webdav as a destination).

I also have a bigger hack of a perl script which attempts to read a 
directory of jars and guess the org, modul, and rev (validating 
against maven2), or prompt the user when it's unclear. See below.

Thanks,
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com


----------------------------------------------
#!/usr/bin/perl
# Looks for jars in the incoming directory, attemps to parse the
# needed ivy info from the filename, and if successful, publishes
# the jar to the Masergy ivrep repository.

# If using linux with package mgmt, dependencies are:
# perl-libwww-perl
# perl-XML-Xpath
# perl-Archive-Zip
use LWP::UserAgent;
use XML::XPath;
use XML::XPath::XMLParser;
use Archive::Zip;
use Cwd;
use strict qw(vars);

use vars qw($indir $ant $rep $pre $post);

$indir = shift(@ARGV);
if ($indir !~ m#^/#) {
   my $cwd = getcwd();
   $indir =~ s/^/$cwd\//;
   print "CWD: $indir\n";
}
$ant = "/usr/local/ant/bin/ant";
$rep = "newjars";
$pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
$post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";

open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
while (<DIR>) {
   my ($org, $module, $jar, $ver);
   next unless /jar/;
   chomp;
   my $file = $indir . "/" . $_;
   $org=$_;
   $module=$_;
   $jar=$_;
   $ver=$_;
   $module=~s/^(.*)\.jar/$1/;
   $module=~s/([^.]*)-\d.*/$1/;
   $ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
   $ver="unknown" unless $ver =~ /\d/;
   ## Figure out the org
   ## Ask the user, if there's ambiguity
   $org = &maven_search_org( $module );
   if (-1 == $org) {
     $org = &guess_org($module);
     print "Org guessed from module name:\n  $org\n";
     $org = &classes_org($file);
     print "Org guessed from jar package naming:\n  $org\n"
       unless (-1 == $org);
     do {
       $org = &get_user_org();
     } while ($org =~ /^$/);
   }
   if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
     print "Skipping jar $jar:\n";
     print "  $org / $module / $ver\n";
     next;
   } else {
     &publish_jar($jar, $org, $module, $ver);
   }
   #&write_ivy_conf($jar, $org, $module, $ver);
}

## Use ivy::install to publish the jar via sftp to ivyrep
sub publish_jar
{
   my ($jar, $org, $module, $ver) = @_;
   print "Uploading $org / $module / $jar ... ";
   my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver 
-Dinstall.path=$indir 2>&1`;
   if ($resp =~ /failed/i) {
     print STDERR "Error uploading $jar:\n", $resp;
     print "\n";
   } else {
     print "Done.\n";
   }
}

## Find a directory in the jar to propose as the org
## searches for any dirs with two path elements
sub classes_org
{
   my ($file) = @_;
   my $somezip = Archive::Zip->new();
   unless ( $somezip->read( $file ) == AZ_OK ) {
        die "Error reading jarfile $file";
   }
   my @members = $somezip->members();
   my @dirnames = ();
   foreach (@members) {
     if ($_->isDirectory()) {
       my $fname = $_->fileName();
       if ($fname =~ /^\w+\/\w+\/$/) {
         $fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
         return $fname;
       }
     }
   }
   return -1;
}

## Ask the user for the best org name
sub get_user_org
{
   print "Enter preffered org:> ";
   my $line = <STDIN>;
   chomp($line);
   return $line;
}

## Make a guess about the org name from the module name
sub guess_org
{
   my ($module) = @_;
   my $org = $module;
     $org=~s/spring.*/springframework/;
     $org=~s/hiber.*/hibernate/;
     $org=~s/standard.*/taglibs/;
     $org=~s/jstl.*/taglibs/;
     $org=~s/persistence.*/sun/;
     $org=~s/jta.*/sun/;
     $org=~s/servlet-api.*/tomcat/;
     $org=~s/ojdbc.*/oracle/;
     $org=~s/cglib.*/cglib/;
     $org=~s/javassist.*/jboss/;
     $org=~s/([^.]*)-\d.*jar/$1/;
     $org=~s/(.*)\.jar/$1/;
   return $org;
}

## Search the maven repo for a module matching this jar
## If found, return the module's organization
## if none, or more than one, are found, ask the user
sub maven_search_org
{
   my ($module) = @_;

   my $ua = LWP::UserAgent->new;
   $ua->agent('Maven Repo Search');

   my $service= 
'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
   my $url=URI->new($service . $module);
   my $req=HTTP::Request->new;
   $req->method('GET');
   $req->uri($url);
   my $resp = $ua->request($req)->content;

   my $org = &parse_resp( $module, $resp );
   return $org;
}

## Parse the XML response from the Maven search service
sub parse_resp
{
   my ($module, $data) = @_;

   my $xp = XML::XPath->new(xml => $data);

   my $nodeset = $xp->find('//groupId'); # find all paragraphs

   # Get rid of duplicates
   my %uniq=();
   foreach my $node ($nodeset->get_nodelist) {
      $uniq{ $xp->getNodeText( $node ) } = 1;
   }
   my @keys = (sort keys %uniq);

   if ($#keys == 0) {
     return ($keys[0]);
   } elsif ($#keys == -1) {
     print "Found no module matching $module in Maven repo\n";
     return -1;
   } else {
     print "Found multiple possible orgs for $module in Maven repo:\n";
     foreach (@keys) {
       print "  $_\n";
     }
     return -1;
   }
}

## (obsolete) Write ant task and ivy.xml conf lines to create repo
sub write_ivy_conf
{
   my ($jar, $org, $module, $ver) = @_;
   my $line = $pre . $org . "\" module=\"" . $module . "\" 
revision=\"" . $ver . "\"". $post . "\n";
   system("mkdir -p $rep/$org/$module/jars");
   link "$indir/$jar", "$rep/$org/$module/jars/$jar";
   print $line;
   print "<dependency org=\"" . $org . "\" name=\"" . $module . "\" 
rev=\"" . $ver . "\"/>\n";
}