ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geißler, Daniel <daniel.geiss...@salt-solutions.de>
Subject AW: Problem when building the same project concurrently for several OSes
Date Wed, 26 Oct 2016 07:50:15 GMT
Hi Misha,

as far as i remember the problem might not be the artifact cache, but the resolution cache.
The behavior I observed was, that ivy keeps some kind of dependency-resolution-cache of recently
resolved artifacts. This cache seems to have no locking and multiple builds may rewrite files
that are currently read by another build.

http://ant.apache.org/ivy/history/2.0.0/settings/caches.html
see the properties for repositoryCacheDir vs. resolutionCacheDir

I haven’t had time to evaluate the behavior, but any feedback is appreciated.

Regards,
Daniel

_____________________________________

SALT Solutions GmbH

Geschäftsführer: Dr. Bernhard Blüthner, Norbert H. Fiedler, Dieter Heyde, Markus Honold
Sitz: München, AG München, HRB 146081 

Hi everybody,

We have some projects, like Zookeeper, that use Ant/Ivy. In some situations such a project
needs to be built multiple times, for different OSes. To save time, we do it on the same machine
and in parallel (we use docker containers to build for different OSes). The main problem in
this scenario is when we want all these builds to share the same Ivy cache, i.e. use it for
concurrent reads and writes.

Ivy 2.4 and artifact-lock-nio locking strategy are supposed to address this. However, when
I attempted to switch to this combination and enable a single cache, my builds failed. I've
verified that NIO-based file locking works across docker containers. I then found and enabled
debug logging in Ivy's FileBasedLockStrategy (one has to invoke ant with '-Divy.log.locking=true').
It looks like locking for artifacts that are downloaded from the web works as expected. However,
in the end there is a failure that says "failed to parse report: /home/cauldron/.ivy2/cache/
org.apache.zookeeper-zookeeper-default.xml". See the end of the log file pasted below. The
file in question contains essentially the list of Zookeeper's external dependencies, and looks
like Ivy itself has just generated it.

So I am confused - why is Ivy trying to read it immediately after writing?
And, generally, is there any chance for us to fix or work around this problem and use the
single, shared Ivy cache?

Regards,

Misha


 [ivy:retrieve] Thread[main,5,main] 1477355454558 entered synchronized area
(locking)
 [ivy:retrieve] Thread[main,5,main] 1477355454558 current status for /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/metadata-3.10.5.Final.ivy.lck
is 1 held locks: Thread[main,5,main]
 [ivy:retrieve] Thread[main,5,main] 1477355454558 reentrant lock acquired on /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/metadata-3.10.5.Final.ivy.lck
in 0ms - hold locks = 2
 [ivy:retrieve] Thread[main,5,main] 1477355454558 releasing lock on /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/
metadata-3.10.5.Final.ivy.lck
 [ivy:retrieve] Thread[main,5,main] 1477355454558 entered synchronized area
(unlocking)
 [ivy:retrieve] Thread[main,5,main] 1477355454558 reentrant lock released on /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/metadata-3.10.5.Final.ivy.lck
- hold locks = 1
 [ivy:retrieve] Thread[main,5,main] 1477355454559 releasing lock on /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/
metadata-3.10.5.Final.ivy.lck
 [ivy:retrieve] Thread[main,5,main] 1477355454559 entered synchronized area
(unlocking)
 [ivy:retrieve] Thread[main,5,main] 1477355454559 lock released on /home/cauldron/.ivy2/cache/io.netty/netty/metadatas/
metadata-3.10.5.Final.ivy.lck
 [ivy:retrieve] :: resolution report :: resolve 468ms :: artifacts dl 131ms
     ---------------------------------------------------------------------
     |                  |            modules            ||   artifacts   |
     |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
     ---------------------------------------------------------------------
     |      default     |   5   |   0   |   0   |   0   ||   5   |   0   |
     ---------------------------------------------------------------------
 [ivy:retrieve] :: retrieving :: org.apache.zookeeper#zookeeper
 [ivy:retrieve]     confs: [default]

 BUILD FAILED
 /container.debian7/output/cdh/zookeeper/zookeeper-3.4.5+cdh6.x+0/build.xml:362:
impossible to ivy retrieve: java.lang.RuntimeException: problem during retrieve of org.apache.zookeeper#zookeeper:
java.text.ParseException:
failed to parse report:
/home/cauldron/.ivy2/cache/org.apache.zookeeper-zookeeper-default.xml:
Premature end of file.
     at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(
RetrieveEngine.java:249)
     at org.apache.ivy.Ivy.retrieve(Ivy.java:561)
     at org.apache.ivy.ant.IvyRetrieve.doExecute(IvyRetrieve.java:98)
     at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:271)
     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
     at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:498)
     at org.apache.tools.ant.dispatch.DispatchUtils.execute(
DispatchUtils.java:106)
     at org.apache.tools.ant.Task.perform(Task.java:348)
     at org.apache.tools.ant.Target.execute(Target.java:390)
     at org.apache.tools.ant.Target.performTasks(Target.java:411)
     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
     at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(
DefaultExecutor.java:41)
     at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
     at org.apache.tools.ant.Main.runBuild(Main.java:809)
     at org.apache.tools.ant.Main.startAnt(Main.java:217)
     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
 Caused by: java.text.ParseException: failed to parse report:
/home/cauldron/.ivy2/cache/org.apache.zookeeper-zookeeper-default.xml:
Premature end of file.
     at org.apache.ivy.plugins.report.XmlReportParser.parse(
XmlReportParser.java:293)
     at org.apache.ivy.core.retrieve.RetrieveEngine.
determineArtifactsToCopy(RetrieveEngine.java:329)
     at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(
RetrieveEngine.java:118)
     ... 19 more
 Caused by: org.xml.sax.SAXParseException; systemId:
file:/home/cauldron/.ivy2/cache/org.apache.zookeeper-zookeeper-default.xml;
lineNumber: 1; columnNumber: 1; Premature end of file.
     at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.
createSAXParseException(ErrorHandlerWrapper.java:203)
     at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.
fatalError(ErrorHandlerWrapper.java:177)
     at com.sun.org.apache.xerces.internal.impl.
XMLErrorReporter.reportError(XMLErrorReporter.java:400)
     at com.sun.org.apache.xerces.internal.impl.
XMLErrorReporter.reportError(XMLErrorReporter.java:327)
     at com.sun.org.apache.xerces.internal.impl.XMLScanner.
reportFatalError(XMLScanner.java:1472)
     at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$
PrologDriver.next(XMLDocumentScannerImpl.java:1014)
     at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(
XMLDocumentScannerImpl.java:602)
     at com.sun.org.apache.xerces.internal.impl.
XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl
.java:505)
     at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(
XML11Configuration.java:841)
     at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(
XML11Configuration.java:770)
     at com.sun.org.apache.xerces.internal.parsers.XMLParser.
parse(XMLParser.java:141)
     at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(
AbstractSAXParser.java:1213)
     at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$
JAXPSAXParser.parse(SAXParserImpl.java:643)
     at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.
parse(SAXParserImpl.java:327)
     at javax.xml.parsers.SAXParser.parse(SAXParser.java:328)
     at org.apache.ivy.plugins.report.XmlReportParser$
SaxXmlReportParser.parse(XmlReportParser.java:249)
     at org.apache.ivy.plugins.report.XmlReportParser.parse(
XmlReportParser.java:291)
     ... 21 more
Mime
View raw message