groovy-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jochen Theodorou <blackd...@gmx.org>
Subject Re: Very Odd/Random JVM Slowdown With Indy
Date Tue, 15 Mar 2016 09:21:09 GMT


On 14.03.2016 22:31, David Clark wrote:
> I've been chasing a slowdown in our application for a couple of months
> now. I have what I believe is a solution (no slowdown for 4 days now).
> But I'm having difficulty understanding why the solution works.
>
> Symptoms:
>
> At random intervals and a random times our web servers will go from
> serving responses in the 300 ms range to taking 30 seconds or more.

30s is pretty bad indeed

> Sometimes the servers will recover, sometimes they require a restart of
> the webserver (spring boot/tomcat). When the applications slow down we
> always see the tomcat thread pool hit the maximum size. Every single
> thread in the thread pool is in the RUNNABLE state but appears to be
> making no progress. Successive thread dumps show that the stacks are
> changing, but VERY slowly. The top of the stack is always this method:
>
> at java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native
> Method).

strange...

> The other common condition is that whatever application code is on the
> stack is always dynamically compiled. Code that is @CompileStatic is
> NEVER on the stack when we see these slowdowns.

static compiled code does not use any invokedynamic yet

> The thread dumps showed that the application code is never waiting on
> locks, socket reads, db connections, etc.

if it is in native code, it might not be showing properly actually.

> Solution:
>
> The solution to the problem was to disable Indy compilation and return
> to non-Indy compilation. However, I don't think Indy is the problem
> here. I noticed that our Spring Boot executable jar contained BOTH
> groovy-all-2.4.5.jar AND groovy-all-indy-2.4.5.jar. Someone forgot to
> exclude the non-indy jars.
>
> My theory:
>
> Having both indy and non-indy jars on the classpath is confusing the JIT
> compiler.

hmm.... I find that unlikely. Either the classes in the jars are defined 
by different class loaders, then they are different classes, or they are 
loaded by the same class loader, in which case the jit still sees only 
one set of classes... which might be from different jars of course, but 
that should not matter... In other words, the origin should not matter

> Code will be continuously JIT-ed as different methods fight
> over which class files to JIT, those loaded from the groovy-all jar or
> those loaded from the groovy-all-indy jar. If this is true then the
> compiler threads will be continuously running and applying native locks
> which are invisible to tools like VisualVM. The result would be random
> slowdowns because only certain combinations of code paths would result
> in slowdowns. It would also cause application code to go very slowly as
> the JIT compiler continuously re-compiles code over and over again.
> Application code would be stuck mostly waiting for JIT operations to
> complete as invalidated code is continuously removed and replaced.
>
> For now I will be leaving Indy disabled until we can do more accurate
> load testing in non production environments.

you know, you can easily test your theory. Remove the non-indy jar from 
the classpath and let the application run with indy. If the slow downs, 
then do not appear anymore you are right.

> My Question:
>
> Is this theory possible? Am I going in a direction that is possible or
> likely?

I find it unlikely, but you never know unless you verify it ;)

I wonder how we can get to the ground of this.

bye Jochen


Mime
View raw message