groovy-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jochen Theodorou <blackd...@gmx.org>
Subject Re: Very Odd/Random JVM Slowdown With Indy
Date Wed, 16 Mar 2016 12:51:29 GMT
Ideal would be to have a small application showing the problem. With an 
actual test we could analyze the problem much better. It is possible the 
issue is related to https://bugs.openjdk.java.net/browse/JDK-8151981, 
but there are also possible bugs in our invokedynamic usage. What I 
could gather so far is that

java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native  Method)

will lock the compiler and stop all threads. And even if jvisualvm shows 
no locking, there might be still many many threads trying to recompile a 
callsite at the same time and wait for a lock to do so.

bye Jochen

On 14.03.2016 22:31, David Clark wrote:
> I've been chasing a slowdown in our application for a couple of months
> now. I have what I believe is a solution (no slowdown for 4 days now).
> But I'm having difficulty understanding why the solution works.
>
> Symptoms:
>
> At random intervals and a random times our web servers will go from
> serving responses in the 300 ms range to taking 30 seconds or more.
> Sometimes the servers will recover, sometimes they require a restart of
> the webserver (spring boot/tomcat). When the applications slow down we
> always see the tomcat thread pool hit the maximum size. Every single
> thread in the thread pool is in the RUNNABLE state but appears to be
> making no progress. Successive thread dumps show that the stacks are
> changing, but VERY slowly. The top of the stack is always this method:
>
> at java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native
> Method).
>
> The other common condition is that whatever application code is on the
> stack is always dynamically compiled. Code that is @CompileStatic is
> NEVER on the stack when we see these slowdowns.
>
> The thread dumps showed that the application code is never waiting on
> locks, socket reads, db connections, etc.
>
> Solution:
>
> The solution to the problem was to disable Indy compilation and return
> to non-Indy compilation. However, I don't think Indy is the problem
> here. I noticed that our Spring Boot executable jar contained BOTH
> groovy-all-2.4.5.jar AND groovy-all-indy-2.4.5.jar. Someone forgot to
> exclude the non-indy jars.
>
> My theory:
>
> Having both indy and non-indy jars on the classpath is confusing the JIT
> compiler. Code will be continuously JIT-ed as different methods fight
> over which class files to JIT, those loaded from the groovy-all jar or
> those loaded from the groovy-all-indy jar. If this is true then the
> compiler threads will be continuously running and applying native locks
> which are invisible to tools like VisualVM. The result would be random
> slowdowns because only certain combinations of code paths would result
> in slowdowns. It would also cause application code to go very slowly as
> the JIT compiler continuously re-compiles code over and over again.
> Application code would be stuck mostly waiting for JIT operations to
> complete as invalidated code is continuously removed and replaced.
>
> For now I will be leaving Indy disabled until we can do more accurate
> load testing in non production environments.
>
> My Question:
>
> Is this theory possible? Am I going in a direction that is possible or
> likely?

Mime
View raw message