groovy-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Paquette <mar...@mac.com>
Subject Re: Very Odd/Random JVM Slowdown With Indy
Date Tue, 15 Mar 2016 14:08:03 GMT
Also, maybe you have checked this already, but if your "code cache" (-XX:ReservedCodeCacheSize)
is too low, it will occur severe slowdowns once this cache is full or near-full.  I would
suggest monitoring this with jconsole or visualvm.

Marc Paquette

> Le 15 mars 2016 à 08:40, Winnebeck, Jason <Jason.Winnebeck@windstream.com> a écrit
:
> 
> Method handles are a relatively new component to the JVM. One thing to consider is whether
or not you are running the latest JVM version.
>  
> If I ignore for a moment your finding of that method on the top of the stack, when I’ve
seen the JVM get very slow without any apparent deadlocks I’ve also seen GC as an issue.
I would just double check to make sure there’s not a leak (in heap or permgen/metaspace)
going on, it might be possible that the MethodHandle code is more susceptible to GC pauses
and appears more often in your stack dumps. Same thing with your underlying OS as well, I
would just double check that you aren’t in a swap condition or (on Linux) in iowait states.
Your note about removing indy fixing this seems to suggest that is not the case but it’s
also a common condition I see with “JVM starts running order of magnitudes slower until
you restart” issues. It may also be possible that Groovy indy or JVM invokedynamic has a
leak that shows up as a memory scaling problem as well.
>  
> Jason
>  
> From: David Clark [mailto:plotinussmith@gmail.com] 
> Sent: Monday, March 14, 2016 5:31 PM
> To: users@groovy.apache.org
> Subject: Very Odd/Random JVM Slowdown With Indy
>  
> I've been chasing a slowdown in our application for a couple of months now. I have what
I believe is a solution (no slowdown for 4 days now). But I'm having difficulty understanding
why the solution works.
>  
> Symptoms:
>  
> At random intervals and a random times our web servers will go from serving responses
in the 300 ms range to taking 30 seconds or more. Sometimes the servers will recover, sometimes
they require a restart of the webserver (spring boot/tomcat). When the applications slow down
we always see the tomcat thread pool hit the maximum size. Every single thread in the thread
pool is in the RUNNABLE state but appears to be making no progress. Successive thread dumps
show that the stacks are changing, but VERY slowly. The top of the stack is always this method:
>  
> at java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native Method).
>  
> The other common condition is that whatever application code is on the stack is always
dynamically compiled. Code that is @CompileStatic is NEVER on the stack when we see these
slowdowns.
>  
> The thread dumps showed that the application code is never waiting on locks, socket reads,
db connections, etc.
>  
> Solution:
>  
> The solution to the problem was to disable Indy compilation and return to non-Indy compilation.
However, I don't think Indy is the problem here. I noticed that our Spring Boot executable
jar contained BOTH groovy-all-2.4.5.jar AND groovy-all-indy-2.4.5.jar. Someone forgot to exclude
the non-indy jars.
>  
> My theory:
>  
> Having both indy and non-indy jars on the classpath is confusing the JIT compiler. Code
will be continuously JIT-ed as different methods fight over which class files to JIT, those
loaded from the groovy-all jar or those loaded from the groovy-all-indy jar. If this is true
then the compiler threads will be continuously running and applying native locks which are
invisible to tools like VisualVM. The result would be random slowdowns because only certain
combinations of code paths would result in slowdowns. It would also cause application code
to go very slowly as the JIT compiler continuously re-compiles code over and over again. Application
code would be stuck mostly waiting for JIT operations to complete as invalidated code is continuously
removed and replaced.
>  
> For now I will be leaving Indy disabled until we can do more accurate load testing in
non production environments.
>  
> My Question:
>  
> Is this theory possible? Am I going in a direction that is possible or likely?
> This email message and any attachments are for the sole use of the intended recipient(s).
Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy all copies of the
original message and any attachments.


Mime
View raw message