ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Weintraub <qazw...@gmail.com>
Subject Re: <ivy:configure> vs. <ivy:settings>
Date Sun, 02 Dec 2012 00:39:51 GMT
> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.

That actually makes a bit more sense. This way, I don't have to go through the rigamarole
of checking the value of the ivy.cleancache property I go through when I do a clean, and I
don't have to depend upon the developers putting the <ivy:cleancache> in their build.xml.

I'm going to see if I can set this property in my ivy.tasks.xml file, so I don't have to redefine
all of my Jenkins jobs. I'll also have to see if this solves the <ivy:configuration>
vs. the <ivy:setitngs> issue which caused this whole thing to come up in the first place.

Thank you for your answer.

On Dec 1, 2012, at 12:04 PM, Mitch Gitman <mgitman@gmail.com> wrote:

> I'm a bit embarrassed by the answer I gave you because it was as recently
> as April that I was actively supporting a Jenkins cluster, and in that
> short time my Jenkins knowledge has slipped a bit. Since I'm going to have
> to do another Jenkins setup myself in about a week, I decided to refresh
> myself a bit about what I'd done before.
> 
> I didn't get very far, but the most important bit of advice I can give you,
> even if it might not be what you want to hear, is this: don't be mapping
> Ivy caches to Jenkins executors.
> 
> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.
> 
> This solves your problem: "This is the problem: Both developers and Jenkins
> use the same build process, and the same build files." The cache will be
> preserved for developers—and trusted for all modules except those that are
> publishing continually changing integration versions. The cache will be
> wiped out every time on the CI server because you've told the CI build
> (unlike the dev build) to run ivy:cleancache, but this is fine because for
> a CI build there's a greater premium on pristineness and reproducibility
> than speed. Because each Jenkins job has its own dedicated workspace and no
> two executors will be running the same job simultaneously, you're assured
> that there will be no concurrency issues between parallel builds.
> 
> One clarification about terminology. The two types of cache are the
> resolution cache and the repository cache.
> 
> If you're not satisfied about the Jenkins build wiping out the repository
> cache every time, there are steps you can take. But get the safe thing
> working first and optimize later. And even then, you should only be
> creating a cache and trusting it for those dependencies (like third-party
> libraries) that never change within the same version.
> 
> On Thu, Nov 29, 2012 at 11:18 PM, David Weintraub <qazwart@gmail.com> wrote:
> 
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>> 
>> That's correct. The environment variable EXECUTOR_NUMBER is set by
>> Jenkins, If you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set.
>> I use that line to set it if it's not already set.
>> 
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>> 
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>> 
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>> 
>> All three of these are true. In fact, when I first used my ivysettings.xml
>> file, I set the executor number manually in my environment. When I did the
>> build, the cache directory was being set correctly. At least when the
>> <ivy:resolve> took place, and the correct cache was being created. I also
>> made sure that the cache was set to $HOME/.ivy2/cache-o when there was no
>> EXECUTOR_NUMBER environment variable. To me, this was the default way the
>> user would run it. Everything looked great.
>> 
>> Suddenly builds on Jenkins started to fail and I figured that happened
>> when parallel builds took place and were wiping out each other's cache.
>> Printing a few debug statements before and after the <ivy:settings> task
>> led me to realize that the cache wasn't being set. That's when I read
>> through the <ivy:settings> task documentation and discovered that
>> <ivy:configure> did more or less the same thing, but that the
>> <ivy:configure> task did the configuration immediately instead of "when
>> needed". Switching to <ivy:configure> fixed the issue.
>> 
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>> 
>> It is. My Ivy project is a subdirectory to the user's project. The
>> developer puts in the lines "<property name="ivy.dir"
>> value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/>
>> into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task
>> for the user. It runs before any other task is called. The user calls
>> <ivy:cleancache> this way:
>> 
>> <property name="ivy.cleancache" value="true"/>
>> 
>> <target name="clean">
>>    <if>
>>       <istrue value="${ivy.cleancache}"/>
>>       <then>
>>           <ivy:cleancache/>
>>       </then>
>>    </if>
>>    <delete>
>>        <fileset dir="${target.dir}"/>
>>    </fileset>
>> </target>
>> 
>> The user can override the value of ${ivy.cleancache} via a
>> build.properties file, so the cache isn't cleaned every time the developer
>> runs the "clean" target. Thus, <ivy:settings/> is called first before
>> <ivy:cleancache/>.
>> 
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>> 
>> The resolution cache needs to be defined so that parallel builds don't
>> interfere with each other. There can also be an issue of a developer using
>> multiple working directories of the same project, and this is also an issue
>> with Jenkins too. The resolution cache gets built pretty quickly, so if you
>> delete it and it has to be built from scratch, it takes maybe another
>> second to do. Thus, putting the resolution cache per checkout resolves the
>> issue of multiple builds sharing the same resolution cache since each will
>> have its own. When the developer does a clean, they delete the "target"
>> directory and the resolution cache. No real problem since it can be quickly
>> rebuilt.
>> 
>> The Ivy cache is a different issue. If this cache is deleted, all the jars
>> must be redownloaded which takes time. Besides if project A downloads
>> version 1.3 of foo.jar, and project B also needs that jar, project B
>> doesn't have to download it. That's the whole purpose of the Ivy cache.
>> Thus, the resolution cache is placed per project while the Ivy jar cache is
>> shared between different projects.
>> 
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>> 
>> No, each job has an established working directory and it uses the same
>> working directory no matter which executor is running. This makes sense
>> since the first time a job might run, it runs on executor #1 and the second
>> time on executor #0. If there were separate working directories per
>> executor, you couldn't take advantage of a version control update which
>> only has to copy a few files over instead of a whole new working directory.
>> 
>> The "cache" really doesn't exist on Jenkins because I wipe it out for each
>> build. That's why each executor needs a separate build. I could just as
>> easily put that under the "target" directory like I do the resolution
>> cache. Unfortunately, the same build.xml is also used by the developer.
>> Their machines are slower and on a slower network connection than the
>> servers. Jenkins can download all of the jars in seconds. On the
>> developer's machines, it can take minutes. Thus, the cache is very
>> important to the developers.
>> 
>> This is the problem: Both developers and Jenkins use the same build
>> process, and the same build files. But, both use the Ivy cache very
>> differently. Developers want it to act like a cache: Saving them valuable
>> time by not having to redownload files over and over. Jenkins wants a clean
>> slate to make sure that there isn't a jar issue during its build process.
>> Satisfying these two completely different requirements while using the same
>> build process and files lead me to this design.
>> 
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
>>> Let me take a stab at addressing your immediate problem, even though I
>>> don't have a direct answer. But then I'm going to suggest a different
>>> approach that avoids that problem.
>>> 
>>> Frankly, I've never used configure, only settings. The configure
>>> documentation explains the primary difference (basically settings is
>> lazy):
>>> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
>>> 
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>> 
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>> 
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>> 
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>> 
>>> And there's something I find odd about your caches specification:
>>>   <caches
>>> 
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>>       resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>> 
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>>> 
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>>> 
>>> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qazwart@gmail.com>
>> wrote:
>>> 
>>>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>>>> 
>>>> I ask because I was having a problem with my Ivy setup. We use
>> Subversion
>>>> and have multiple projects setup that use Ant. In order to centralize
>> our
>>>> jar dependencies, I've decided to implement Ivy into our process.
>>>> 
>>>> I did this by creating an Ivy project (
>> https://github.com/qazwart/ivy.dir)
>>>> in Subversion. Users merely set `svn:externals` to include this project,
>>>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>>>> and everything is set. I control the ivysettings.xml file, so I point
>> Ivy
>>>> to our repository. The Ivy jar is part of the Ivy project, so users
>> don't
>>>> have to install it. The idea was to make Ivy integration as painless as
>>>> possible.
>>>> 
>>>> We use Jenkins as our build system, and I decided it would be good if we
>>>> could clean the Ivy cache before each build. However, out Jenkins system
>>>> has six executors, so as many as six builds could happen at once.
>> Cleaning
>>>> the cache in one build while another one runs could cause problems, so I
>>>> modified my ivysettings.xml file to use a different cache depending upon
>>>> the executor:
>>>> 
>>>> <ivysettings>
>>>>   <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>>>   <caches
>>>> 
>>>> 
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>>>       resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>>>   <settings defaultResolver="default"/>
>>>>   <include file="${ivy.dir}/ivysettings-public.xml"/>
>>>>   <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>>>>   <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>>>>   <include
>> url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>>>>   <include
>>>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>>>> </ivysettings>
>>>> 
>>>> The problem is that if I did an <ivy:cleancache>, the cache ID was
>> always
>>>> set to "cache-0". When I switched from "<ivy:settings>" to
>>>> "<ivy:configure>", the problem went away. (In the standard build
>> process,
>>>> the <ivy:cleancache> task is executed before the <ivy:resolve>
task).
>>>> 
>>>> I'm told that "<ivy:settings>" can do multiple configurations, but
it
>> looks
>>>> like <ivy:configure> also can do multiple settings too. I also read
that
>>>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>>>> that in the on line documentation.
>>>> 
>>>> So what is the difference between <ivy:settings> and <ivy:configure>,
>> and
>>>> when should I use one over the other?
>>>> 
>>>> --
>>>> David Weintraub
>>>> qazwart@gmail.com
>>>> 
>> 
>> 


Mime
View raw message