ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maarten Coene <maarten_co...@yahoo.com>
Subject Re: Running Ivy in parallel builds
Date Fri, 20 Mar 2009 11:01:31 GMT

Hi,

I think you can solve it like this
- use the same cache for your artifacts
- use a different (per build?) cache for your resolution data
- use the "artifact-lock" locking strategy on your artifact cache

More info about the caches is here:
http://ant.apache.org/ivy/history/latest-release/settings/caches.html

Maarten




----- Original Message ----
From: Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>
To: ivy-user@ant.apache.org
Sent: Friday, March 20, 2009 11:45:21 AM
Subject: RE: Running Ivy in parallel builds

OMG, this is totally unacceptable. We started to use Ivy to optimize space
for builds and automatize build tools setups. My Ivy cache stores about
600Mb of different tools initially which are reused then for every build. So
using cache per build is a nightmare! Using cache per running parallel build
sounds better but developing this custom configuration can make a lot of
unexpected problems afterthen.
Ivy on his own should know if cache is used by another process and wait for
it. Does anybody is going to implement this functionality? Since I`m using
special Ivy wrapper I can try to do it on my own on my side but it is not a
good idea anyway.

Regards,
Uldis, C. T. Co
uldis.karlovs-karlovskis@ctco.lv


-----Original Message-----
From: Martin Eigenbrodt [mailto:martineigenbrodt@googlemail.com] 
Sent: ceturtdiena, 2009. gada 19. martā 20:31
To: ivy-user@ant.apache.org
Subject: Re: Running Ivy in parallel builds

I do run Ivy on Hudson and use one Cache per "Executor". So builds in
parallel have different caches but sequential builds may share
a cache. Hudson provides an Environment Variabel with the Executor Number
that I use within the ivysettings.xml to set different Cachedirs. A similair
setup may be possible in other environments.
A second concurency issue is that  the buildnumber and deploy tasks are
seperated. There should be an atomic "deployWithNext Buildnumber" or you
will run in problems when the same project is build in parallel.

Best regards,

Martin2009/3/19 Sam Berlin <sberlin@gmail.com>

> Yes, I have noticed significant problems when using ivy in a parallel
> build environment (essentially, any build-server), and we've only had
> Ivy in use on the buildserver for ~12 hours.  The only solution I can
> think of is to make the cache local per build, which is pretty far
> from ideal.
>
> Any suggestions for how to make Ivy more robust against concurrent builds?
>
> Sam
>
> 2009/3/19 Uldis Karlovs-Karlovskis <uldis.karlovs-karlovskis@ctco.lv>:
> > And this is not all. Previous case was when artifact is already in
cache.
> If
> > Ivy needs to download new artifact then parallel execution gives
> something
> > like this:
> >
> > ---------------------------------------------------------------------
> >        | | modules || artifacts |
> >        | conf | number| search|dwnlded|evicted|| number|dwnlded|
> >
> > ---------------------------------------------------------------------
> >        | default | 4 | 0 | 0 | 0 || 6 | 0 |
> >
> > ---------------------------------------------------------------------
> >
> >  :: problems summary ::
> >  :::: WARNINGS
> >        problem while downloading module descriptor:
> >
>
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
> > Type=xml: impossible to move part file to definitive one:
> > /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original.part ->
> > /root/.ivy2/cache/ctco/somemodule/ivy-1.3.1.16.xml.original (21ms)
> >
> >         module not found: ctco#somemodule;1.3.1.16
> >
> >        ==== bones-repository: tried
> >
> >
> >
>
http://server/download?name=somemodule&version=1.3.1.16&file=somemodule&file
> > Type=xml
> >
> >         ::::::::::::::::::::::::::::::::::::::::::::::
> >
> >         :: UNRESOLVED DEPENDENCIES ::
> >
> >         ::::::::::::::::::::::::::::::::::::::::::::::
> >
> >         :: ctco#somemodule;1.3.1.16: not found
> >
> >         ::::::::::::::::::::::::::::::::::::::::::::::
> >
> >
> >
> >  :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> >  command exit code: 1
> >
> > Regards,
> > Uldis, C. T. Co
> > uldis.karlovs-karlovskis@ctco.lv
> >
> >
> > -----Original Message-----
> > From: Uldis Karlovs-Karlovskis [mailto:uldis.karlovs-karlovskis@ctco.lv]
> > Sent: ceturtdiena, 2009. gada 19. martā 17:37
> > To: ivy-user@ant.apache.org
> > Subject: Running Ivy in parallel builds
> >
> > Hello!
> >
> >
> >
> > Have You tested running Ivy in more than one build at the time?
> >
> > I see there are few files in cache which got updated every time I run
> Ivy.
> > When my build system is running more than one process at the time
> sometimes
> > I got this exception:
> >
> > :: retrieving :: lv.ctco#scm-bootstrap
> >
> >  confs: [default]
> >
> >  Caught: java.lang.RuntimeException: problem during retrieve of
> > <namespace>#<module>: java.text.ParseException: failed to parse report:
> > /root/.ivy2/cache/lv.ctco-scm-bootstrap-default.xml: Premature end of
> file.
> >
> >  command exit code: 1
> >
> >
> >
> >
> >
> > Uldis Karlovs - Karlovskis
> >
> > Lead SCM Engineer
> >
> > System Configuration Management
> >
> > C.T.Co
> >
> > Mobile: +371 29 345 210
> >
> >
> >
> >
> >
>


      

Mime
View raw message