groovy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain Stalder <astal...@span.ch>
Subject Re: Groovy Grape/Ivy thread safe?
Date Tue, 28 Apr 2015 19:15:43 GMT
Hi again,

Looks like everybody is busy with stuff on a somewhat lower level of 
Maslov's Pyramid? ;)

So, I will simply answer myself to my questions:

 > It looks to me like Grape resp. ivy is not thread safe.
 >
 > Question 1: Is that a bug or just how things currently are? (If it's 
a bug, rather Grape bug or an ivy bug, resp. where would I best file it?)

I consider that a bug, even though it seems to me that fixing it might 
possibly be not so simple...

I created an account for the Groovy Apache JIRA, resp. seems like my 
user had been already transferred there (or maybe I created an account 
in the past and forgotten about it), with the intent to file a bug with 
Groovy, but apparently I don't have the rights to do so?

If its possible without jumping through too many hoops, I would still be 
available to file the bug (how?), else this post will have to suffice... ;)

 > Question 2: If it's not a bug, is that a general issue, i.e. if I 
compile two sets of sources with the GroovyCompiler (say I create a 
CompilationUnit instance and add sources) will there be similar issues 
if both sets of sources have the same Grape dependencies? (Yes, I know, 
I could just try, but maybe the answer obvious to someone who knows the 
implementation.)

Yes, it is definitely a general issue, see unit test further below which 
uses CompilationUnit directly.

Question 3 (new): Am I stuck?

No, I am not, at least not with Grengine: http://grengine.ch

I can simply override DefaultGroovyCompiler with a class that 
synchronizes around CompilationUnit.compile(), since Grengine is not 
using GroovyClassLoader, GroovyShell or GroovyScriptEngine, except a 
GroovyClassLoader during compilation. (Of course, in order to use Grape, 
a GroovyClassLoader has to be in the classpath at runtime (or a 
RootLoader, actually is there any real reason for that?), but that is 
also possible to do by configuring Grengine.)

Or is there a workaround that works with Groovy itself, on the level of 
the compiler or GroovyClassLoader/GroovyShell/GroovyScriptEngine, except 
synchronizing each an every call that might compile anything?

If not, I will probably enhance Grengine to make the workaround simpler, 
independently of whether this will be fixed in the future, in order to 
provide a workaround for older Groovy versions.

Best wishes,
Alain

Here is the Java unit test:

import static org.junit.Assert.assertTrue;

import java.util.Collections;
import java.util.LinkedList;
import java.util.List;

import org.codehaus.groovy.control.CompilationUnit;
import org.codehaus.groovy.control.Phases;
import org.junit.Test;

public class GroovyCompileConcurrencyTest {

     @Test
     public void testConcurrencyLowLevel() throws Exception {

         final List<Throwable> errors = Collections.synchronizedList(new 
LinkedList<Throwable>());

         final int nLoops = 1000;
         final int nThreads = 10;

         for (int i=0; i<nLoops; i++) {

             System.out.print(".");

             List<Thread> scriptThreads = new LinkedList<Thread>();

             for (int j=0; j<nThreads; j++) {
                 final CompilationUnit cu = new CompilationUnit();
                 cu.addSource("Whatever",
                         "@Grab('com.google.guava:guava:18.0')\n" +
                         "import com.google.common.base.Ascii\n");

                 Thread scriptThread = new Thread(
                         new Runnable() {
                             public void run() {
                                 try {
cu.compile(Phases.CLASS_GENERATION);
                                 } catch (Throwable t) {
                                     errors.add(t);
                                     t.printStackTrace();
                                 }
                             }
                         });
                 scriptThreads.add(scriptThread);
             }

             for (Thread scriptThread : scriptThreads) {
                 scriptThread.start();
             }

             for (Thread scriptThread : scriptThreads) {
                 scriptThread.join();
             }

             assertTrue("must be true", errors.size() == 0);
         }

     }

}


On 26.04.15 17:52, Alain Stalder wrote:
> Hi there,
>
> In the Java unit test listed further below, I create 100 GroovyShell 
> instances and add the same directory to the classpath of the 
> GroovyClassLoader of each GroovyShell. This directory contains 100 
> script files, named Util0.groovy to Util99.groovy, containing the 
> following script (with XX replaced by 0..99):
>
> @Grab('com.google.guava:guava:18.0')
> import com.google.common.base.Ascii
> class UtilXX {
>   static boolean isUpperCase(def c) {
>     return Ascii.isUpperCase(c)\n"
>   }
> }
>
> The unit test then runs
>
>     shell.evaluate("return Util" + j + ".isUpperCase('C' as char)");
>
> in 100 separate threads (with j from 0..99).
>
> This test does not always fail, but often. Most of the time, a 
> ConcurrentModificationException occurs down in ivy (which is used by 
> Grape). In other cases, a MultipleCompilationErrorsException ocurred 
> at "@Grab('com.google.guava:guava:18.0')". (Full stacktraces at the 
> bottom.)
>
> It looks to me like Grape resp. ivy is not thread safe.
>
> Question 1: Is that a bug or just how things currently are? (If it's a 
> bug, rather Grape bug or an ivy bug, resp. where would I best file it?)
>
> Question 2: If it's not a bug, is that a general issue, i.e. if I 
> compile two sets of sources with the GroovyCompiler (say I create a 
> CompilationUnit instance and add sources) will there be similar issues 
> if both sets of sources have the same Grape dependencies? (Yes, I 
> know, I could just try, but maybe the answer obvious to someone who 
> knows the implementation.)
>
> (I have tested this with Groovy 2.4.3 and ivy 2.4.0 on MacOS X with 
> Java 6, but also I have also seen the same issues on CentOS with 
> several Groovy/ivy versions and Java 7, on a deployed webapp, so it 
> seems to be fairly independent of the exact environment.)
>
> Best wishes,
>
> Alain
>
>
> -------------------------------------
> Java Unit Test (does not always fail)
> -------------------------------------
>
> import static org.junit.Assert.assertEquals;
> import static org.junit.Assert.assertTrue;
> import groovy.lang.GroovyShell;
>
> import java.io.File;
> import java.io.FileNotFoundException;
> import java.io.PrintWriter;
> import java.io.UnsupportedEncodingException;
> import java.util.LinkedList;
> import java.util.List;
>
> import org.junit.Before;
> import org.junit.Rule;
> import org.junit.Test;
> import org.junit.rules.TemporaryFolder;
>
> public class GrapeAndGroovyShellConcurrencyTest {
>
>     @Rule
>     public TemporaryFolder tempFolder = new TemporaryFolder();
>
>     private volatile boolean testPassed;
>
>     @Before
>     public void setUp() {
>         testPassed = true;
>     }
>
>     public static void setFileText(File file, String text)
>             throws FileNotFoundException, UnsupportedEncodingException {
>         PrintWriter writer = new PrintWriter(file, "UTF-8");
>         writer.write(text);
>         writer.close();
>     }
>
>     @Test
>     public void testConcurrency() throws Exception {
>
>         File dir = tempFolder.getRoot();
>         final int n = 100;
>         for (int i=0; i<n; i++) {
>             File f1 = new File(dir, "Util" + i + ".groovy");
>             setFileText(f1, "@Grab('com.google.guava:guava:18.0')\n"
>                     + "import com.google.common.base.Ascii\n"
>                     + "class Util" + i + " {\n"
>                     + "  static boolean isUpperCase(def c) {\n"
>                     + "    return Ascii.isUpperCase(c)\n"
>                     + "  }\n"
>                     + "}\n"
>                     );
>         }
>
>         List<Thread> scriptThreads = new LinkedList<Thread>();
>         for (int i=0; i<n; i++) {
>             final GroovyShell shell = new GroovyShell();
> shell.getClassLoader().addClasspath(dir.getAbsolutePath());
>
>             final int j = i;
>
>             Thread scriptThread = new Thread(
>                     new Runnable() {
>                         public void run() {
>                             try {
>                                 assertEquals(true, 
> shell.evaluate("return Util" + j + ".isUpperCase('C' as char)"));
>                                 System.out.println("Thread " + 
> Thread.currentThread().getName() + " : OK");
>                             } catch (Throwable t) {
>                                 System.out.println("Thread " + 
> Thread.currentThread().getName() + " : FAILED");
>                                 t.printStackTrace();
>                                 testPassed = false;
>                             }
>                         }
>                     });
>             scriptThread.setDaemon(true);
>             scriptThread.setName("run-" + i);
>             scriptThread.start();
>             scriptThreads.add(scriptThread);
>         }
>
>         for (Thread scriptThread : scriptThreads) {
>             scriptThread.join();
>         }
>
>         assertTrue("must be true", testPassed);
>     }
>
> }
>
> ------------
> Stacktrace 1
> ------------
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: 
> startup failed:
> General error during conversion: 
> java.util.ConcurrentModificationException
>
> java.util.ConcurrentModificationException
>     at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
>     at java.util.ArrayList$Itr.next(ArrayList.java:831)
>     at 
> org.apache.ivy.util.MessageLoggerHelper.sumupProblems(MessageLoggerHelper.java:45)
>     at 
> org.apache.ivy.util.MessageLoggerEngine.sumupProblems(MessageLoggerEngine.java:136)
>     at org.apache.ivy.util.Message.sumupProblems(Message.java:143)
>     at 
> org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:347)
>     at org.apache.ivy.Ivy.resolve(Ivy.java:523)
>     at org.apache.ivy.Ivy$resolve$1.call(Unknown Source)
>     at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:404)
>     at sun.reflect.GeneratedMethodAccessor671.invoke(Unknown Source)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at 
> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166)
>     at 
> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
>     at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:563)
>     at groovy.grape.GrapeIvy$resolve$56.callCurrent(Unknown Source)
>     at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:532)
>     at groovy.grape.GrapeIvy$resolve$45.callCurrent(Unknown Source)
>     at 
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
>     at 
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151)
>     at 
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:179)
>     at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:254)
>     at groovy.grape.Grape.grab(Grape.java:163)
>     at 
> groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:358)
>     at 
> org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:319)
>     at 
> org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:928)
>     at 
> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
>     at 
> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:566)
>     at 
> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:543)
>     at 
> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
>     at 
> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
>     at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
>     at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
>     at 
> GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
>     at java.lang.Thread.run(Thread.java:745)
>
> 1 error
>
>     at 
> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
>     at 
> org.codehaus.groovy.control.ErrorCollector.addException(ErrorCollector.java:155)
>     at 
> org.codehaus.groovy.control.SourceUnit.addException(SourceUnit.java:345)
>     at 
> groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:367)
>     at 
> org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:319)
>     at 
> org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:928)
>     at 
> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
>     at 
> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:566)
>     at 
> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:543)
>     at 
> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
>     at 
> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
>     at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
>     at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
>     at 
> GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
>     at java.lang.Thread.run(Thread.java:745)
>
> ------------
> Stacktrace 2
> ------------
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: 
> startup failed:
> file:/var/folders/38/r0n49vmn7zg5dffk79_tgpl80000gn/T/junit124846036508580912/Util7.groovy:

> 1: unable to resolve class com.google.common.base.Ascii
>  @ line 1, column 1.
>    @Grab('com.google.guava:guava:18.0')
>    ^
>
> 1 error
>
>     at 
> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
>     at 
> org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
>     at 
> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
>     at 
> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:539)
>     at 
> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
>     at 
> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
>     at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
>     at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
>     at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
>     at 
> GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
>     at java.lang.Thread.run(Thread.java:745)
> .
>


Mime
View raw message