ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uldis Karlovs-Karlovskis" <uldis.karlovs-karlovs...@ctco.lv>
Subject RE: Running Ivy in parallel builds
Date Tue, 24 Mar 2009 07:11:48 GMT
Yes, of course! That`s what I`m saying. Using different resolution caches on
parallel builds works much better but it fails anyway, if there is being
downloaded new artifact from repository (to default cache). So
lockStrategy=artifact-lock doesn`t help.

Regards,
Uldis, C. T. Co
uldis.karlovs-karlovskis@ctco.lv
 

-----Original Message-----
From: Sam Berlin [mailto:sberlin@gmail.com] 
Sent: piektdiena, 2009. gada 20. martā 18:05
To: ivy-user@ant.apache.org
Subject: Re: Running Ivy in parallel builds

Even with artifact locking enabled, the resolution cache *must* be
unique per build if concurrent builds are occurring.

Sam

On Fri, Mar 20, 2009 at 11:50 AM, Uldis Karlovs-Karlovskis
<uldis.karlovs-karlovskis@ctco.lv> wrote:
> The parallel artifacts download unfortunately failed. Artifact-lock is not
helping. However, this is much better than it was.
>
> Regards,
> Uldis, C. T. Co
> uldis.karlovs-karlovskis@ctco.lv
>
>
> -----Original Message-----
> From: Alan Chaney [mailto:alan@writingshow.com]
> Sent: piektdiena, 2009. gada 20. martā 17:47
> To: ivy-user@ant.apache.org
> Subject: Re: Running Ivy in parallel builds
>
> Hi Maarten
> Maarten Coene wrote:
>> Hi,
>>
>> I think you can solve it like this
>> - use the same cache for your artifacts
>> - use a different (per build?) cache for your resolution data
>> - use the "artifact-lock" locking strategy on your artifact cache
>>
>> More info about the caches is here:
>> http://ant.apache.org/ivy/history/latest-release/settings/caches.html
>>
>> Maarten
>>
>>
> I hope you don't consider the following too off-topic but I'm trying to
> set up a parallel build and I'm a little confused about the docs as they
> relate to caching. In
> http://ant.apache.org/ivy/history/latest-release/concept.html#cache
> it states:
> While there is always only one resolution cache, you can define multiple
> repository caches
> <http://ant.apache.org/ivy/history/latest-release/settings/caches.html>,
> each resolver
> <http://ant.apache.org/ivy/history/latest-release/settings/resolvers.html>
> being able to use a separate cache.
>
> But in the actual caches/cache description it says that you can define
> the cachedir within your ivysettings.xml. I assume that the 'only one
> resolution cache' means 'per ivy settings'. Is this correct?
>
> If you have multiple ivy settings defined in a project and you specify
> which settings you use you can presumably use different resolution
> caches for different aspects of the project?
>
> Regards
>
> Alan Chaney
>
>
>
>>
>>
>> ----- Original Message ----
>> From: Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>
>> To: ivy-user@ant.apache.org
>> Sent: Friday, March 20, 2009 11:45:21 AM
>> Subject: RE: Running Ivy in parallel builds
>>
>> OMG, this is totally unacceptable. We started to use Ivy to optimize
space
>> for builds and automatize build tools setups. My Ivy cache stores about
>> 600Mb of different tools initially which are reused then for every build.
So
>> using cache per build is a nightmare! Using cache per running parallel
build
>> sounds better but developing this custom configuration can make a lot of
>> unexpected problems afterthen.
>> Ivy on his own should know if cache is used by another process and wait
for
>> it. Does anybody is going to implement this functionality? Since I`m
using
>> special Ivy wrapper I can try to do it on my own on my side but it is not
a
>> good idea anyway.
>>
>> Regards,
>> Uldis, C. T. Co
>> uldis.karlovs-karlovskis@ctco.lv
>>
>>
>> -----Original Message-----
>> From: Martin Eigenbrodt [mailto:martineigenbrodt@googlemail.com]
>> Sent: ceturtdiena, 2009. gada 19. marta 20:31
>> To: ivy-user@ant.apache.org
>> Subject: Re: Running Ivy in parallel builds
>>
>> I do run Ivy on Hudson and use one Cache per "Executor". So builds in
>> parallel have different caches but sequential builds may share
>> a cache. Hudson provides an Environment Variabel with the Executor Number
>> that I use within the ivysettings.xml to set different Cachedirs. A
similair
>> setup may be possible in other environments.
>> A second concurency issue is that  the buildnumber and deploy tasks are
>> seperated. There should be an atomic "deployWithNext Buildnumber" or you
>> will run in problems when the same project is build in parallel.
>>
>> Best regards,
>>
>> Martin2009/3/19 Sam Berlin <sberlin@gmail.com>
>>
>>
>>> Yes, I have noticed significant problems when using ivy in a parallel
>>> build environment (essentially, any build-server), and we've only had
>>> Ivy in use on the buildserver for ~12 hours.  The only solution I can
>>> think of is to make the cache local per build, which is pretty far
>>> from ideal.
>>>
>>> Any suggestions for how to make Ivy more robust against concurrent
builds?
>>>
>>> Sam
>>>
>>> 2009/3/19 Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>:
>>>
>>>> And this is not all. Previous case was when artifact is already in
>>>>
>> cache.
>>
>>> If
>>>
>>>> Ivy needs to download new artifact then parallel execution gives
>>>>
>>> something
>>>
>>>> like this:
>>>>
>>>> ---------------------------------------------------------------------
>>>>        | | modules || artifacts |
>>>>        | conf | number| search|dwnlded|evicted|| number|dwnlded|
>>>>
>>>> ---------------------------------------------------------------------
>>>>        | default | 4 | 0 | 0 | 0 || 6 | 0 |
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>  :: problems summary ::
>>>>  :::: WARNINGS
>>>>        problem while downloading module descriptor:
>>>>
>>>>
>>
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
>>
>>>> Type=xml: impossible to move part file to definitive one:
>>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original.part ->
>>>> /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original (21ms)
>>>>
>>>>         module not found: ctco#somemodule;1.3.1.16
>>>>
>>>>        ==== bones-repository: tried
>>>>
>>>>
>>>>
>>>>
>>
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
>>
>>>> Type=xml
>>>>
>>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>>
>>>>         :: UNRESOLVED DEPENDENCIES ::
>>>>
>>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>>
>>>>         :: ctco#somemodule;1.3.1.16: not found
>>>>
>>>>         ::::::::::::::::::::::::::::::::::::::::::::::
>>>>
>>>>
>>>>
>>>>  :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>>>>  command exit code: 1
>>>>
>>>> Regards,
>>>> Uldis, C. T. Co
>>>> uldis.karlovs-karlovskis@ctco.lv
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Uldis Karlovs-Karlovskis
[mailto:uldis.karlovs-karlovskis@ctco.lv]
>>>> Sent: ceturtdiena, 2009. gada 19. marta 17:37
>>>> To: ivy-user@ant.apache.org
>>>> Subject: Running Ivy in parallel builds
>>>>
>>>> Hello!
>>>>
>>>>
>>>>
>>>> Have You tested running Ivy in more than one build at the time?
>>>>
>>>> I see there are few files in cache which got updated every time I run
>>>>
>>> Ivy.
>>>
>>>> When my build system is running more than one process at the time
>>>>
>>> sometimes
>>>
>>>> I got this exception:
>>>>
>>>> :: retrieving :: lv.ctco#scm-bootstrap
>>>>
>>>>  confs: [default]
>>>>
>>>>  Caught: java.lang.RuntimeException: problem during retrieve of
>>>> <namespace>#<module>: java.text.ParseException: failed to parse
report:
>>>> /root/.ivy2/cache/lv.ctco-scm-bootstrap-default.xml: Premature end of
>>>>
>>> file.
>>>
>>>>  command exit code: 1
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Uldis Karlovs - Karlovskis
>>>>
>>>> Lead SCM Engineer
>>>>
>>>> System Configuration Management
>>>>
>>>> C.T.Co
>>>>
>>>> Mobile: +371 29 345 210
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>>
>>
>> !DSPAM:49c3796450102136417547!
>>
>>
>
>
>
>



Mime
View raw message