groovy-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ion Alberdi (JIRA)" <>
Subject [jira] [Commented] (GROOVY-8097) Add an argument to set the resolution cache path in @Grab
Date Tue, 07 Mar 2017 09:05:33 GMT


Ion Alberdi commented on GROOVY-8097:

To give more context on our use case:  we use grab to download up-to-date versions of libraries
that implement part of the computation of thousands of concurrent jenkins jobs. The rate of
updates reached for now a peak of 19 new versions a single day.

[~paulk] Switching to something else than Ivy may indeed be more adapted to use cases similar
to ours. For now, during the updates done in a month, at least two concurrent job write to
the same resolution cache dir simultaneously. In this scenario, the systemGroovyScripts relying
on these libraries are broken as the grab executed on the master's JVM uses a corrupted resolutionCacheDir.
In this scenario, we have no choice but rebooting the master.

[~jexler] The concurrency issue is not between resolutionCacheDir and repositoryCache dir.
The repositoryCache dir is thread-safe (as Ivy permits to put a lock on repositoryCache dirs).
Let's call cJob1, cJob2 two concurrent jobs. The issue is between the resolutionCacheDir of
cJob1 and cJob2 that are executed in parallel. We'd like to provide an API to make
- cJob1 use repositoryCacheDir, resolutionCacheDir1
- cJob2 use repositoryCacheDir, resolutionCacheDir2

Providing we create and upload thousands of different grapeConfig1.xml, ..., grapeConfig999.xml
and using the grape.config system property you suggested, we will be able to make for ex,
cJob2 use resolutionCacheDir2. I however see two drawbacks:
- quite a lot of duplicated grabeConfig.xml, they will indeed all be the same (only the resolutionCacheDir
will be different),
- put a dependency between the code that deploys grapeConfig.xml-s and the code that deploys
new jobs: if 1000 new jobs should be added, 1000 new files should be generated/deployed.

Regarding processes/Java VMs, issues within a VM, we have both use cases
- grab done in groovyCommand, are in the between process/Java VMs use case
- grab done in systemGroovyCommands, are in the same VM.

More precisely seems impacted by the same
limitation of Ivy. When reading, 
"If Groovy scripts that import the same libraries via Grape are compiled in separate threads,
compilation may fail due to race conditions."
my guess is that if the two threads use a locked repositoryCacheDir and a different resolutionCacheDir,
the bug will not occur.

> Add an argument to set the resolution cache path in @Grab
> ---------------------------------------------------------
>                 Key: GROOVY-8097
>                 URL:
>             Project: Groovy
>          Issue Type: New Feature
>          Components: Grape
>    Affects Versions: 2.4.8
>            Reporter: Ion Alberdi
>            Priority: Minor
> Ivy does not support concurrent access to its resolution cache 
> Grape relies on Ivy. For this reason, Grape cannot support concurrent access to its resolution
cache neither.
> When using the @Grab annotation in jenkins groovyCommand or systemGroovyCommand, the
related code is vulnerable to race conditions. When the race condition appears in a systemGroovyCommand,
we have no choice but to reboot jenkins as all consecutive calls to @Grab fail.
> Among the two solutions we tried: 
> - Protect the calls to grab with a lock similar to ivy's "artifact-lock-nio" strategy.
Works but slow.
> - Set Ivy's lock on the repository cache and setup Grab to use a different cache resolution
cache for each concurrent jobs. The following code permits to fix a test we did to reproduce
the race condition.
> {code}
>     static IvySettings createIvySettings(String resolutionPath, boolean dumpSettings)
>         // Copy/Paste/Purged from GrapeIvy.groovy
>         IvySettings settings = new IvySettings()
>         settings.load(new File(GROOVY_HOME, "grapeConfig.xml"))
>         // set up the cache dirs
>         settings.defaultCache = new File(GRAPES_HOME)
>         settings.setVariable("ivy.default.configuration.m2compatible", "true")
>         settings.setDefaultResolutionCacheBasedir(resolutionPath)
>         return settings
>     }
>     static GrapeIvy ivyWithCustomResolutionPath(String resolutionPath) {
>         Class<?> grapeIvyClass = Class.forName("groovy.grape.GrapeIvy");
>         Object instance = grapeIvyClass.newInstance()
>         Field field = grapeIvyClass.getDeclaredField("ivyInstance");
>         field.setAccessible(true);
>         field.set(instance, Ivy.newInstance(createIvySettings(resolutionPath)));
>         return ((GrapeIvy)instance)
>     }
> {code}
> We'd like to propose to add an additional argument to Grab to setup Ivy's resolution
cache directory.
> Note that this solution seems to have been adopted by these users too
> Would you agree on such a feature ? We'd be glad to propose a PR.

This message was sent by Atlassian JIRA

View raw message