ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Weintraub <qazw...@gmail.com>
Subject Re: <ivy:configure> vs. <ivy:settings>
Date Fri, 30 Nov 2012 06:18:04 GMT
On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
>  <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>

That's correct. The environment variable EXECUTOR_NUMBER is set by Jenkins, If you're not
running on Jenkins, then, EXECUTOR_NUMBER isn't set. I use that line to set it if it's not
already set.

> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
> 
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
> 
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.

All three of these are true. In fact, when I first used my ivysettings.xml file, I set the
executor number manually in my environment. When I did the build, the cache directory was
being set correctly. At least when the <ivy:resolve> took place, and the correct cache
was being created. I also made sure that the cache was set to $HOME/.ivy2/cache-o when there
was no EXECUTOR_NUMBER environment variable. To me, this was the default way the user would
run it. Everything looked great.

Suddenly builds on Jenkins started to fail and I figured that happened when parallel builds
took place and were wiping out each other's cache. Printing a few debug statements before
and after the <ivy:settings> task led me to realize that the cache wasn't being set.
That's when I read through the <ivy:settings> task documentation and discovered that
<ivy:configure> did more or less the same thing, but that the <ivy:configure>
task did the configuration immediately instead of "when needed". Switching to <ivy:configure>
fixed the issue.

> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.

It is. My Ivy project is a subdirectory to the user's project. The developer puts in the lines
"<property name="ivy.dir" value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/>
into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task for the user.
It runs before any other task is called. The user calls <ivy:cleancache> this way:

<property name="ivy.cleancache" value="true"/>

<target name="clean">
    <if>
       <istrue value="${ivy.cleancache}"/>
       <then>
           <ivy:cleancache/>
       </then>
    </if>
    <delete>
        <fileset dir="${target.dir}"/>
    </fileset>
</target>

The user can override the value of ${ivy.cleancache} via a build.properties file, so the cache
isn't cleaned every time the developer runs the "clean" target. Thus, <ivy:settings/>
is called first before <ivy:cleancache/>.

> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.

The resolution cache needs to be defined so that parallel builds don't interfere with each
other. There can also be an issue of a developer using multiple working directories of the
same project, and this is also an issue with Jenkins too. The resolution cache gets built
pretty quickly, so if you delete it and it has to be built from scratch, it takes maybe another
second to do. Thus, putting the resolution cache per checkout resolves the issue of multiple
builds sharing the same resolution cache since each will have its own. When the developer
does a clean, they delete the "target" directory and the resolution cache. No real problem
since it can be quickly rebuilt.

The Ivy cache is a different issue. If this cache is deleted, all the jars must be redownloaded
which takes time. Besides if project A downloads version 1.3 of foo.jar, and project B also
needs that jar, project B doesn't have to download it. That's the whole purpose of the Ivy
cache. Thus, the resolution cache is placed per project while the Ivy jar cache is shared
between different projects.

> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.

No, each job has an established working directory and it uses the same working directory no
matter which executor is running. This makes sense since the first time a job might run, it
runs on executor #1 and the second  time on executor #0. If there were separate working directories
per executor, you couldn't take advantage of a version control update which only has to copy
a few files over instead of a whole new working directory.

The "cache" really doesn't exist on Jenkins because I wipe it out for each build. That's why
each executor needs a separate build. I could just as easily put that under the "target" directory
like I do the resolution cache. Unfortunately, the same build.xml is also used by the developer.
Their machines are slower and on a slower network connection than the servers. Jenkins can
download all of the jars in seconds. On the developer's machines, it can take minutes. Thus,
the cache is very important to the developers.

This is the problem: Both developers and Jenkins use the same build process, and the same
build files. But, both use the Ivy cache very differently. Developers want it to act like
a cache: Saving them valuable time by not having to redownload files over and over. Jenkins
wants a clean slate to make sure that there isn't a jar issue during its build process. Satisfying
these two completely different requirements while using the same build process and files lead
me to this design.

On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
> Let me take a stab at addressing your immediate problem, even though I
> don't have a direct answer. But then I'm going to suggest a different
> approach that avoids that problem.
> 
> Frankly, I've never used configure, only settings. The configure
> documentation explains the primary difference (basically settings is lazy):
> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
> 
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
>  <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> 
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
> 
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
> 
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
> 
> And there's something I find odd about your caches specification:
>    <caches
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>        resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> 
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
> 
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
> 
> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qazwart@gmail.com> wrote:
> 
>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>> 
>> I ask because I was having a problem with my Ivy setup. We use Subversion
>> and have multiple projects setup that use Ant. In order to centralize our
>> jar dependencies, I've decided to implement Ivy into our process.
>> 
>> I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
>> in Subversion. Users merely set `svn:externals` to include this project,
>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>> and everything is set. I control the ivysettings.xml file, so I point Ivy
>> to our repository. The Ivy jar is part of the Ivy project, so users don't
>> have to install it. The idea was to make Ivy integration as painless as
>> possible.
>> 
>> We use Jenkins as our build system, and I decided it would be good if we
>> could clean the Ivy cache before each build. However, out Jenkins system
>> has six executors, so as many as six builds could happen at once. Cleaning
>> the cache in one build while another one runs could cause problems, so I
>> modified my ivysettings.xml file to use a different cache depending upon
>> the executor:
>> 
>> <ivysettings>
>>    <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>    <caches
>> 
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>        resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>    <settings defaultResolver="default"/>
>>    <include file="${ivy.dir}/ivysettings-public.xml"/>
>>    <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>>    <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>>    <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>>    <include
>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>> </ivysettings>
>> 
>> The problem is that if I did an <ivy:cleancache>, the cache ID was always
>> set to "cache-0". When I switched from "<ivy:settings>" to
>> "<ivy:configure>", the problem went away. (In the standard build process,
>> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>> 
>> I'm told that "<ivy:settings>" can do multiple configurations, but it looks
>> like <ivy:configure> also can do multiple settings too. I also read that
>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>> that in the on line documentation.
>> 
>> So what is the difference between <ivy:settings> and <ivy:configure>,
and
>> when should I use one over the other?
>> 
>> --
>> David Weintraub
>> qazwart@gmail.com
>> 


Mime
View raw message