cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5345) Potential problem with GarbageCollectorMXBean
Date Wed, 25 Jun 2014 21:31:28 GMT


Joshua McKenzie commented on CASSANDRA-5345:

[~enigmacurry] - did you or your team have any luck reproducing this?

It should be trivial to throw a gc.isValid() check in the logGCResults loop and if invalid,
flag to rebuild the List<GarbageCollectorMXBean> we're iterating across in that function
as well as to introduce some exception handling for the UndeclaredThrowableException.  I'm
not finding much on the logic behind *when* these MXBeans can be removed from the system and
I agree with Brandon that it seems incredibly odd for a JVM to punt and re-init a garbage
collector MXBean on the fly with such infrequency that we haven't seen this more often.

The fact that Arya had a nodetool failure connecting to the Memory subsystem:
{code:title=Memory failure}
Exception in thread "main" java.lang.IllegalArgumentException:
is concerning as it would seem to imply some deeper problems w/the JMX integration of the
Memory subsystem on these JVM's given that we're querying the factory by String there rather
than trying to use an invalid reference.

> Potential problem with GarbageCollectorMXBean
> ---------------------------------------------
>                 Key: CASSANDRA-5345
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.7
>         Environment: JVM:JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_30
 typical 6 node 2 availability zone Mutli DC cluster on linux vms with
> and mx4j-tools.jar and jna.jar both on path. Default configuration bar token setup(equispaced),
sensible file and use of said snitch.
>            Reporter: Matt Byrd
>            Assignee: Joshua McKenzie
> I am not certain this is definitely a bug, but I thought it might be worth posting to
see if someone with more JVM//JMX knowledge could disprove my reasoning. Apologies if I've
failed to understand something.
> We've seen an intermittent problem where there is an uncaught exception in the scheduled
task of logging gc results in
> {code}
> ...
>  ERROR [ScheduledTasks:1] 2013-03-08 01:09:06,335 (line
139) Fatal exception in thread Thread[ScheduledTasks:1,5,main]
> java.lang.reflect.UndeclaredThrowableException
>         at $Proxy0.getName(Unknown Source)
>         at org.apache.cassandra.service.GCInspector.logGCResults(
>         at org.apache.cassandra.service.GCInspector.access$000(
>         at org.apache.cassandra.service.GCInspector$
>         at java.util.concurrent.Executors$
>         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(
>         at java.util.concurrent.FutureTask.runAndReset(
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> Caused by: java.lang:name=ParNew,type=GarbageCollector
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(
>         at com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(
>         at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(
>         at
>         ... 13 more
> ...
> {code}
> I think the problem, may be caused by the following reasoning:
> In GcInspector we populate a list of mxbeans when the GcInspector instance is instantiated:
> {code}
> ...
> List<GarbageCollectorMXBean> beans = new ArrayList<GarbageCollectorMXBean>();
>         MBeanServer server = ManagementFactory.getPlatformMBeanServer();
>         try
>         {
>             ObjectName gcName = new ObjectName(ManagementFactory.GARBAGE_COLLECTOR_MXBEAN_DOMAIN_TYPE
+ ",*");
>             for (ObjectName name : server.queryNames(gcName, null))
>             {
>                 GarbageCollectorMXBean gc = ManagementFactory.newPlatformMXBeanProxy(server,
name.getCanonicalName(), GarbageCollectorMXBean.class);
>                 beans.add(gc);
>             }
>         }
>         catch (Exception e)
>         {
>             throw new RuntimeException(e);
>         }
> ...
> {code}
> Cassandra then periodically calls:
> {code}
> ...
>     private void logGCResults()
>     {
>         for (GarbageCollectorMXBean gc : beans)
>         {
>             Long previousTotal = gctimes.get(gc.getName());
> ...
> {code}
> In the oracle javadocs, they seem to suggest that these beans could disappear at any
time.(I'm not sure why when or how this might happen)
> See: getGarbageCollectorMXBeans
> {code}
> ...
> public static List<GarbageCollectorMXBean> getGarbageCollectorMXBeans()
> Returns a list of GarbageCollectorMXBean objects in the Java virtual machine. The Java
virtual machine may have one or more GarbageCollectorMXBean objects. It may add or remove
GarbageCollectorMXBean during execution.
> Returns:
> a list of GarbageCollectorMXBean objects.
> ...
> {code}
> Correct me if I'm wrong, but do you think this might be causing the problem? That somehow
the JVM decides to remove the GarbageCollectorMXBean temporarily or permanently (causing said
exception) and if this is expected behaviour, should it be handled in some way?
> Also I'd like to point out that this may be an issue on other versions as well as I don't
believe this code has changed in quite a long time.
> Unfortunately I haven't been able to reproduce this outside of the production environment,
if you have any tips, questions or are able to explain//disprove my concerns, I'd be very
> Thanks,
> Matt

This message was sent by Atlassian JIRA

View raw message