groovy-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Clark <plotinussm...@gmail.com>
Subject Re: Very Odd/Random JVM Slowdown With Indy
Date Wed, 16 Mar 2016 19:31:50 GMT
If I can think of a way to reproduce the problem with a small script or
application I will definitely post it to the list. I really have no idea
what would cause this problem. If you have any suggestions about what types
of application behavior would tend to lead to this issue I will give them a
try.

Thank for the link to the JDK bug, that does seem to be what we are seeing.

On Wed, Mar 16, 2016 at 7:51 AM, Jochen Theodorou <blackdrag@gmx.org> wrote:

> Ideal would be to have a small application showing the problem. With an
> actual test we could analyze the problem much better. It is possible the
> issue is related to https://bugs.openjdk.java.net/browse/JDK-8151981, but
> there are also possible bugs in our invokedynamic usage. What I could
> gather so far is that
>
> java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native
> Method)
>
> will lock the compiler and stop all threads. And even if jvisualvm shows
> no locking, there might be still many many threads trying to recompile a
> callsite at the same time and wait for a lock to do so.
>
> bye Jochen
>
> On 14.03.2016 22:31, David Clark wrote:
>
>> I've been chasing a slowdown in our application for a couple of months
>> now. I have what I believe is a solution (no slowdown for 4 days now).
>> But I'm having difficulty understanding why the solution works.
>>
>> Symptoms:
>>
>> At random intervals and a random times our web servers will go from
>> serving responses in the 300 ms range to taking 30 seconds or more.
>> Sometimes the servers will recover, sometimes they require a restart of
>> the webserver (spring boot/tomcat). When the applications slow down we
>> always see the tomcat thread pool hit the maximum size. Every single
>> thread in the thread pool is in the RUNNABLE state but appears to be
>> making no progress. Successive thread dumps show that the stacks are
>> changing, but VERY slowly. The top of the stack is always this method:
>>
>> at java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(Native
>> Method).
>>
>> The other common condition is that whatever application code is on the
>> stack is always dynamically compiled. Code that is @CompileStatic is
>> NEVER on the stack when we see these slowdowns.
>>
>> The thread dumps showed that the application code is never waiting on
>> locks, socket reads, db connections, etc.
>>
>> Solution:
>>
>> The solution to the problem was to disable Indy compilation and return
>> to non-Indy compilation. However, I don't think Indy is the problem
>> here. I noticed that our Spring Boot executable jar contained BOTH
>> groovy-all-2.4.5.jar AND groovy-all-indy-2.4.5.jar. Someone forgot to
>> exclude the non-indy jars.
>>
>> My theory:
>>
>> Having both indy and non-indy jars on the classpath is confusing the JIT
>> compiler. Code will be continuously JIT-ed as different methods fight
>> over which class files to JIT, those loaded from the groovy-all jar or
>> those loaded from the groovy-all-indy jar. If this is true then the
>> compiler threads will be continuously running and applying native locks
>> which are invisible to tools like VisualVM. The result would be random
>> slowdowns because only certain combinations of code paths would result
>> in slowdowns. It would also cause application code to go very slowly as
>> the JIT compiler continuously re-compiles code over and over again.
>> Application code would be stuck mostly waiting for JIT operations to
>> complete as invalidated code is continuously removed and replaced.
>>
>> For now I will be leaving Indy disabled until we can do more accurate
>> load testing in non production environments.
>>
>> My Question:
>>
>> Is this theory possible? Am I going in a direction that is possible or
>> likely?
>>
>

Mime
View raw message