ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Chaney <a...@writingshow.com>
Subject Re: Running Ivy in parallel builds
Date Fri, 20 Mar 2009 15:47:08 GMT
Hi Maarten
Maarten Coene wrote:
> Hi,
>
> I think you can solve it like this
> - use the same cache for your artifacts
> - use a different (per build?) cache for your resolution data
> - use the "artifact-lock" locking strategy on your artifact cache
>
> More info about the caches is here:
> http://ant.apache.org/ivy/history/latest-release/settings/caches.html
>
> Maarten
>
>   
I hope you don't consider the following too off-topic but I'm trying to 
set up a parallel build and I'm a little confused about the docs as they 
relate to caching. In 
http://ant.apache.org/ivy/history/latest-release/concept.html#cache
it states:
While there is always only one resolution cache, you can define multiple 
repository caches 
<http://ant.apache.org/ivy/history/latest-release/settings/caches.html>, 
each resolver 
<http://ant.apache.org/ivy/history/latest-release/settings/resolvers.html> 
being able to use a separate cache.

But in the actual caches/cache description it says that you can define 
the cachedir within your ivysettings.xml. I assume that the 'only one 
resolution cache' means 'per ivy settings'. Is this correct?

If you have multiple ivy settings defined in a project and you specify 
which settings you use you can presumably use different resolution 
caches for different aspects of the project?

Regards

Alan Chaney



>
>
> ----- Original Message ----
> From: Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>
> To: ivy-user@ant.apache.org
> Sent: Friday, March 20, 2009 11:45:21 AM
> Subject: RE: Running Ivy in parallel builds
>
> OMG, this is totally unacceptable. We started to use Ivy to optimize space
> for builds and automatize build tools setups. My Ivy cache stores about
> 600Mb of different tools initially which are reused then for every build. So
> using cache per build is a nightmare! Using cache per running parallel build
> sounds better but developing this custom configuration can make a lot of
> unexpected problems afterthen.
> Ivy on his own should know if cache is used by another process and wait for
> it. Does anybody is going to implement this functionality? Since I`m using
> special Ivy wrapper I can try to do it on my own on my side but it is not a
> good idea anyway.
>
> Regards,
> Uldis, C. T. Co
> uldis.karlovs-karlovskis@ctco.lv
>
>
> -----Original Message-----
> From: Martin Eigenbrodt [mailto:martineigenbrodt@googlemail.com] 
> Sent: ceturtdiena, 2009. gada 19. martā 20:31
> To: ivy-user@ant.apache.org
> Subject: Re: Running Ivy in parallel builds
>
> I do run Ivy on Hudson and use one Cache per "Executor". So builds in
> parallel have different caches but sequential builds may share
> a cache. Hudson provides an Environment Variabel with the Executor Number
> that I use within the ivysettings.xml to set different Cachedirs. A similair
> setup may be possible in other environments.
> A second concurency issue is that  the buildnumber and deploy tasks are
> seperated. There should be an atomic "deployWithNext Buildnumber" or you
> will run in problems when the same project is build in parallel.
>
> Best regards,
>
> Martin2009/3/19 Sam Berlin <sberlin@gmail.com>
>
>   
>> Yes, I have noticed significant problems when using ivy in a parallel
>> build environment (essentially, any build-server), and we've only had
>> Ivy in use on the buildserver for ~12 hours.  The only solution I can
>> think of is to make the cache local per build, which is pretty far
>> from ideal.
>>
>> Any suggestions for how to make Ivy more robust against concurrent builds?
>>
>> Sam
>>
>> 2009/3/19 Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>:
>>     
>>> And this is not all. Previous case was when artifact is already in
>>>       
> cache.
>   
>> If
>>     
>>> Ivy needs to download new artifact then parallel execution gives
>>>       
>> something
>>     
>>> like this:
>>>
>>> ---------------------------------------------------------------------
>>>        | | modules || artifacts |
>>>        | conf | number| search|dwnlded|evicted|| number|dwnlded|
>>>
>>> ---------------------------------------------------------------------
>>>        | default | 4 | 0 | 0 | 0 || 6 | 0 |
>>>
>>> ---------------------------------------------------------------------
>>>
>>>  :: problems summary ::
>>>  :::: WARNINGS
>>>        problem while downloading module descriptor:
>>>
>>>       
> http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
>   
>>> Type=xml: impossible to move part file to definitive one:
>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original.part ->
>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original (21ms)
>>>
>>>         module not found: ctco#somemodule;1.3.1.16
>>>
>>>        ==== bones-repository: tried
>>>
>>>
>>>
>>>       
> http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
>   
>>> Type=xml
>>>
>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>
>>>         :: UNRESOLVED DEPENDENCIES ::
>>>
>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>
>>>         :: ctco#somemodule;1.3.1.16: not found
>>>
>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>
>>>
>>>
>>>  :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>>>  command exit code: 1
>>>
>>> Regards,
>>> Uldis, C. T. Co
>>> uldis.karlovs-karlovskis@ctco.lv
>>>
>>>
>>> -----Original Message-----
>>> From: Uldis Karlovs-Karlovskis [mailto:uldis.karlovs-karlovskis@ctco.lv]
>>> Sent: ceturtdiena, 2009. gada 19. martā 17:37
>>> To: ivy-user@ant.apache.org
>>> Subject: Running Ivy in parallel builds
>>>
>>> Hello!
>>>
>>>
>>>
>>> Have You tested running Ivy in more than one build at the time?
>>>
>>> I see there are few files in cache which got updated every time I run
>>>       
>> Ivy.
>>     
>>> When my build system is running more than one process at the time
>>>       
>> sometimes
>>     
>>> I got this exception:
>>>
>>> :: retrieving :: lv.ctco#scm-bootstrap
>>>
>>>  confs: [default]
>>>
>>>  Caught: java.lang.RuntimeException: problem during retrieve of
>>> <namespace>#<module>: java.text.ParseException: failed to parse report:
>>> /root/.ivy2/cache/lv.ctco-scm-bootstrap-default.xml: Premature end of
>>>       
>> file.
>>     
>>>  command exit code: 1
>>>
>>>
>>>
>>>
>>>
>>> Uldis Karlovs - Karlovskis
>>>
>>> Lead SCM Engineer
>>>
>>> System Configuration Management
>>>
>>> C.T.Co
>>>
>>> Mobile: +371 29 345 210
>>>
>>>
>>>
>>>
>>>
>>>       
>
>
>       
>
>
> !DSPAM:49c3796450102136417547!
>
>   


Mime
View raw message