ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Tharp <joshua-th...@alumni.calpoly.edu>
Subject Re: Building a little bit more advanced repository, any tutorial or information out there?
Date Fri, 08 May 2009 18:36:42 GMT
I've basically done what you are talking about with a combination of
CruiseControl and IvyRoundup (http://code.google.com/p/ivyroundup/). (I'm
cross-posting to IvyRoundup in case anyone there finds this useful or might
be able to provide more input.) I have my CruiseControl machine checkout a
read-only version of IvyRoundup. It does periodic updates to capture any
changes made to the IvyRoundup repository. I then took portions of the build
script that comes with IvyRoundup, and created my own build script that
stages their source packages, overlays the packages I've added, resolves and
retrieves the packages that we use, and publishes them via a filesystem
publisher to a directory on our build box that is served up by apache.

Roundup uses a packager resolver so instead of ivy.xml files and jars, it
provides ivy.xml and packager.xml files. The jars remain at their original
websites or in the Maven repository (that is until they have been resolved).

I put my additions in our SVN repository as the directory IvyRepo. I used
SVN external and mapped http://ivyroundup.googlecode.com/svn/trunk/src to
the directory reposrc, so after a svn co of IvyRepo the directory looks like
this:
build.xml (shown below)
filter.properties (shown below)
reposrc (SVN external)
local-modules (my own additions)
|_ javax.javaee
|_ org.w3c.css
|_ ...

You'll notice that in my ivysettings, I don't ever just go out on the
Internet to resolve stuff. This way, we can make whatever is available in
IvyRoundup available, or we can weed out packages that we don't want to have
internally, we can also add our own 3rd party packages with ivy.xml and
packager.xml files.

Here are some files to show what I'm talking about.
====================================
My build.xml executed by CruiseControl
====================================
<?xml version="1.0" encoding="UTF-8"?>

<!--
 ! Builds the local Ivy repository using the IvyRoundup project as the seed
 ! -->
<project
    name="IvyRepo"
    default="update"
    basedir="."
    xmlns:ivy="urn:org.apache.ivy.ant">

<!-- This is from where apache serves up our repository -->
<property
    name="pub.repo"
    location="/var/www/html/repository" />

<property
    name="src.d"
    value="reposrc" />
<property
    name="src.ad"
    location="${src.d}" />
<property
    name="build.d"
    value="build" />
<property
    name="build.ad"
    location="${build.d}" />
<property
    name="stage.d"
    value="stage" />
<property
    name="stage.ad"
    location="stage" />
<property
    name="modules.file"
    location="${pub.repo}/modules.xml" />
<property
    name="resolve.d"
    value="resolve" />

<target
    name="update"
    description="Updates the local Ivy Repo."
    depends="clean,
             prep,
             repo,
             resolve" />

<target name="clean">
  <delete dir="${stage.d}" />
  <delete dir="cache" />
  <delete dir="${resolve.d}" />
  <delete dir="${build.d}" />
  <delete
      dir="${pub.repo}"
      includeEmptyDirs="true">
    <exclude name="modules/com.mycompany*/**" /> <!-- This is where
CruiseControl is publishing our own build artifacts for team sharing so we
don't want to delete our own artifacts. -->
    <include name="css/**" />
    <include name="images/**" />
    <include name="modules.xml" />
    <include name="xsd/**" />
    <include name="xsl/**" />
    <include name="**/ivy.xml*" />
    <include name="**/packager.xml*" />
  </delete>
</target>

<target
    name="prep"
    depends="prep:init,
             prep:mkdirs" />

<target name="prep:init">
  <property
      name="rev.cmd"
      value="( svn info ${src.d} 2>/dev/null
               | grep -E '^Last Changed Rev:' || echo '0' )
               | sed 's/^Last Changed Rev: //g'" />
  <exec
      outputproperty="svn.revision"
      executable="/bin/bash"
      logError="true"
      failonerror="true">
    <arg value="-c" />
    <arg value="${rev.cmd}" />
  </exec>

  <property file="filter.properties" />
</target>

<target name="prep:mkdirs">
  <mkdir dir="${stage.d}" />
  <mkdir dir="${build.d}" />
  <mkdir dir="${resolve.d}" />
</target>

<target
    name="repo"
    depends="repo:copy,
             repo:modules,
             repo:xform" />

<target name="repo:copy">
  <copy todir="${stage.d}">
    <!-- Fist fileset is to get all the supporting files -->
    <fileset dir="${src.d}">
      <include name="images/**" />
      <include name="xsl/ivy-doc.xsl" />
      <include name="xsl/modules.xsl" />
      <include name="css/ivy-style.css" />
      <include name="xsd/packager-1.0.xsd" />
      <include name="xsd/ivy.xsd" />
    </fileset>
    <!-- Second fileset is to get all the downloaded modules -->
    <fileset dir="${src.d}">
      <include name="modules/**/*" />
    </fileset>
  </copy>
  <copy
      todir="${stage.d}/modules"
      overwrite="true">
    <!-- Third fileset is to get the local additions -->
    <fileset dir="local-modules">
      <include name="**/*" />
    </fileset>
  </copy>

  <copy todir="${pub.repo}">
    <fileset dir="${stage.d}">
      <exclude name="ivy.xml" />
      <exclude name="packager.xml" />
    </fileset>
  </copy>
</target>

<target name="repo:modules">
  <property
      name="gen.mod.cmd"
      value="TIMESTAMP=`date -u +'%Y-%m-%d %H:%M:%S %Z'` ;
             export LC_ALL=C;
             find ${stage.d}/modules -name ivy.xml -print
                 | sort
                 | awk -F/ -v SVNREV=${svn.revision}
                     -v TIMESTAMP=&quot;$TIMESTAMP&quot;
                     -f ${src.ad}/awk/xmlify.awk
                 > ${modules.file}" />
  <exec
      executable="/bin/bash"
      logError="true"
      failonerror="true">
    <arg value="-c" />
    <arg value="${gen.mod.cmd}" />
  </exec>
</target>

<target
    name="repo:xform"
    depends="xform:svnsub,
             xform:ivy,
             xform:packager">
  <!-- Validate all generated XML files -->
  <xmlvalidate
      failonerror="true"
      warn="yes"
      lenient="no">
    <attribute
        name="http://xml.org/sax/features/namespaces"
        value="true"/>
    <attribute
        name="http://apache.org/xml/features/validation/schema"
        value="true"/>
    <attribute
        name="http://apache.org/xml/features/validation/schema-full-checking
"
        value="true"/>
    <fileset dir="${pub.repo}/modules">
      <include name="**/ivy.xml"/>
      <include name="**/packager.xml"/>
    </fileset>
  </xmlvalidate>
</target>

<target name="xform:svnsub">
  <copy
      file="${src.d}/xsl/ivy-repo.xsl"
      todir="${build.d}/xsl">
    <filterset>
      <filter
          token="SVNROOTPATH"
          value="${ivyroundup.svn.url}" />
    </filterset>
  </copy>
</target>

<target name="xform:ivy">
  <!-- Generate ivy.xml files -->
  <xslt
      style="${src.d}/xsl/apply-xsl.xsl"
      in="${modules.file}"
      out="/dev/null"
      force="true">
    <param
        name="filename"
        expression="ivy.xml"/>
    <param
        name="stylesheet"
        expression="${build.ad}/xsl/ivy-repo.xsl"/>
    <param
        name="target.repo"
        expression="${pub.repo}" />
    <param
        name="source.base"
        expression="${stage.ad}" />
  </xslt>
</target>

<target name="xform:packager">
  <!-- Generate packager.xml files -->
  <xslt
      style="${src.d}/xsl/apply-xsl.xsl"
      in="${modules.file}"
      out="/dev/null"
      force="true">
    <param
        name="filename"
        expression="packager.xml"/>
    <param
        name="stylesheet"
        expression="packager-repo.xsl"/>
    <param
        name="target.repo"
        expression="${pub.repo}" />
    <param
        name="source.base"
        expression="${stage.ad}" />
  </xslt>
</target>

<target
    name="resolve"
    depends="resolve:mkbuild,
             resolve:do" />

<target name="resolve:mkbuild">
  <xslt
      style="${src.d}/xsl/resolve.xsl"
      in="${modules.file}/"
      out="${resolve.d}/build.xml">
    <param
        name="retrieve.pattern"

expression="${pub.repo}/modules/[organisation]/[module]/[revision]/[type]s/[artifact]-[revision](-[classifier]).[ext]"
/>
    <param
        name="resolve.org"
        expression="${resolve.org}" />
    <param
        name="resolve.mod"
        expression="${resolve.mod}" />
    <param
        name="resolve.rev"
        expression="${resolve.rev}" />
  </xslt>

  <copy
      file="${src.d}/ivy/resolve-settings.xml"
      todir="${resolve.d}" />
</target>

<target name="resolve:do">
  <ant
      dir="${resolve.d}"
      inheritall="false">
    <property
        name="target.repo"
        value="${pub.repo}" />
  </ant>
</target>

</project>

====================================
filter.properties (comma separated lists will determine which packages get
retrieved, all empty will resolve everything)
====================================
resolve.org=org.apache.commons,org.apache.ant
resolve.mod=
resolve.rev=

====================================
ivysettings.xml
====================================
<?xml version="1.0" encoding="UTF-8"?>
<ivysettings>
  <settings defaultResolver="default" />

  <property
      name="ivy.local.default.root"
      value="${ivy.default.ivy.user.dir}/local"
      override="false" />
  <property
      name="ivy.local.default.ivy.pattern"
      value="[organisation]/[module]/[revision]/ivy.xml"
      override="false" />
  <property
      name="ivy.local.default.artifact.pattern"
      value="[organisation]/[module]/[revision]/[type]s/[module].[ext]"
      override="false" />

  <property
      name="module.pat"
      value="modules/[organisation]/[module]/[revision]" />
  <property
      name="ivy.pat"
      value="ivy.xml" />
  <property
      name="packager.pat"
      value="packager.xml" />
  <property
      name="artifact.pat"
      value="[type]s/[artifact]-[revision](-[classifier]).[ext]" />
  <property
      name="repo.ivy.pat"
      value="${module.pat}/${ivy.pat}" />
  <property
      name="repo.packager.pat"
      value="${module.pat}/${packager.pat}" />
  <property
      name="repo.artifact.pat"
      value="${module.pat}/${artifact.pat}" />

  <property
      name="repo.host"
      value="http://myserver" />
  <property
      name="repo.root"
      value="${repo.host}/repository" />

  <resolvers>
    <filesystem
        name="local"
        checkmodified="true"
        changingMatcher="regexp"
        changingPattern=".*_SNAPSHOT">
      <ivy pattern="${ivy.local.default.root}/${repo.ivy.pat}" />
      <artifact pattern="${ivy.local.default.root}/${repo.artifact.pat}" />
    </filesystem>

    <url name="shared">
      <ivy pattern="${repo.root}/${repo.ivy.pat}" />
      <artifact pattern="${repo.root}/${repo.artifact.pat}" />
    </url>

    <filesystem
        name="smg-repository"
        local="true">
      <ivy pattern="${ivy.publish.d}/${repo.ivy.pat}" />
      <artifact pattern="${ivy.publish.d}/${repo.artifact.pat}" />
    </filesystem>

    <packager
        name="public"
        buildRoot="${ivy.default.ivy.user.dir}/packager/build"
        resourceCache="${ivy.default.ivy.user.dir}/packager/cache">
      <ivy pattern="${repo.root}/${repo.ivy.pat}" />
      <artifact pattern="${repo.root}/${repo.packager.pat}" />
    </packager>
  </resolvers>

  <!-- <include url="${ivy.default.settings.dir}/ivysettings-public.xml" />
-->
  <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml" />
  <include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"
/>
</ivysettings>



On Fri, May 8, 2009 at 9:08 AM, Garima Bathla <garima.bathla@gmail.com>wrote:

> OK; Seems like a scenario which should be very common? Most of the teams
> will not want to depend on public repository all the time - what if  a
> developer wants to work offline with  no internet access without messing
> around with settings?
>
> I am no IVY expert - but below is a sure shot solution that my colleague
> has
> implemented for our team to work in similar model - go to public repository
> only when it is needed.
>
> So we have 3 types of repositories:
>
> 1. Thirdparty repositories - where we publish third-party jars that do not
> exist on any public repository - we convert them to an ivy module and
> publish.
> 2. Internal repository - where we publish all our teams specific modules .
> 3. public repository - used to resolve the rest of the modules ( we depend
> on spring source repo).
>
> So in your case first of all if you can build company repository
> successfully then I assume you will want Ivy to resolve in below order
>
> 1. Try  Company repository ( with ibiblio modules and thirdparty modules?)
> 2. Try ibiblio repository
>
> So in your ivysettings you can define new resolvers i.e.
>
> Shared resolver using company repository to resolve modules
> Public resolver using ibiblio to resolve modules
>
> and then chain them
>
> <chain name="chain-resolver" returnFirst="true"
>   <resolver ref="shared-resolver"
>  <resolver ref="public-resolver">
> </chain>
>
> So you will have to modify ivysettings.xml and not ivy.xml.  I hope you do
> have ivysettings.xml?
>
> HTH,
>
> Garima.
>
>
>
>
> On Fri, May 8, 2009 at 4:25 AM, Kent Larsson <kent.larsson@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have an ivy.xml which looks something like: (for proper formatting:
> > http://pastebin.com/f21948262 )
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <ivy-module version="2.0"
> >  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >  xsi:noNamespaceSchemaLocation="
> http://ant.apache.org/ivy/schemas/ivy.xsd
> > ">
> >        <info
> >          organisation="net.mycompany"
> >          module="webcollab"
> >          status="integration"/>
> >        <configurations>
> >                <conf name="runtime" description="" />
> >                <conf name="compile" extends="runtime" description="" />
> >        </configurations>
> >        <dependencies>
> >                <dependency org="org.apache.wicket" name="wicket"
> > rev="1.3.5"
> > conf="runtime->default;foride->sources,javadoc">
> >                        <exclude org="org.apache.portals.bridges"
> > module="portals-bridges-common"/>
> >                        <exclude org="javax.portlet"
> module="portlet-api"/>
> >                </dependency>
> >                <dependency org="junit" name="junit" rev="4.5"
> > conf="runtime->default;foride->sources,javadoc" />
> >                <dependency org="org.apache.wicket"
> > name="wicket-spring-annot"
> > rev="1.3.5" conf="runtime->default;foride->sources,javadoc"/>
> >                <dependency org="org.slf4j" name="slf4j-log4j12"
> rev="1.5.6"
> > conf="runtime->default;foride->sources,javadoc"/>
> >                <dependency org="org.slf4j" name="jcl-over-slf4j"
> > rev="1.5.6"
> > conf="runtime->default;foride->sources,javadoc"/>
> >                <!-- Some dependencies removed -->
> >        </dependencies>
> > </ivy-module>
> >
> > My current ivy.xml uses the default repository ibiblio in Maven2-mode.
> >
> > What I would like is:
> >
> > 1. Have a company repository with copies of everything I usually take
> > from ibiblio.
> >
> > 2. Also have some JAR's in that repository which are not in ibiblio,
> > like JavaMail and Activation, both from Sun.
> >
> > 3. In the Ivy resolution on the client machines, try the company
> > repository first, then try ibiblio.
> >
> > Motivation for 3 is that if I add something to my ivy.xml which is
> > available in ibiblio I could get it directly, while my company
> > repository might be synchronized with ibiblio once each day. Also if
> > my company repository is down it might be nice to fall back on ibiblio
> > for non licensed files (and hope that JavaMail etc is in the local
> > cache already).
> >
> > Is there any information out there for this scenario? It seems like it
> > would be a common scenario. I can't find any, and I don't know where
> > to start. If I just can get some more information I'll try to create a
> > blog post about it or something, in case there is a lack of
> > information on the net.
> >
> > Thank you for reading! I hope you can help me out. Have a nice weekend!
> >
> > Best regards, Kent
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message