cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Ribeiro (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-11823) Creating a table leads to a race with GraphiteReporter
Date Tue, 02 Aug 2016 22:05:20 GMT


Edward Ribeiro updated CASSANDRA-11823:
    Attachment: CASSANDRA-11823.patch

Hi [~ostefano] and [~Stefania], 

I took a stab at this issue, and I guess I've found the root cause of the problem. I am providing
a patch for cassandra-3.0 branch.

*IMHO*, it looks like when a table is created, the metrics Set for a specific key entry at
{{TableMetrics.allTableMetrics}} is updated while the metrics {{Set}} is being iterated to
get a summarized value to be passed to {{GraphiteReporter}}, as below, for example:

            public Long getValue()
                long total = 0;
                for (Metric cfGauge : allTableMetrics.get(name))
                    total = total + ((Gauge<? extends Number>) cfGauge).getValue().longValue();
                return total;

Even tough {{allTableMetrics}} is a thread-safe {{ConcurrentMap}}, *the {{Set}} iterated in
the for-loop above is not!* Oddly enough, the  {{ConcurrentModificationException}} reports
the {{Map}} as the offending one instead of the {{Set}} inside the {{Map}} that's effectively
being iterated (I guess that is is due to the nature of the for-each loop).

*If this is the case*, the solution is to create a thread-safe {{Set}}.  {{Collections#synchronizedSet}}
will not work, but fortunately, we can also  create a thread-safe {{Set}} backed by a {{ConcurrentHashMap}}.
Until Java 8, we could do this as shown here:

But as C* uses Java 8 this can be done as here:

Of course, I can be chasing my own tail (would not the first time, lol) and the problem has
*nothing* to do with I exposed above, so, please, let me know what you think. :)

> Creating a table leads to a race with GraphiteReporter
> ------------------------------------------------------
>                 Key: CASSANDRA-11823
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Stefano Ortolani
>            Priority: Minor
>              Labels: lhf
>         Attachments: CASSANDRA-11823.patch
> Happened only on 3/4 nodes out of 13.
> {code:xml}
> INFO  [MigrationStage:1] 2016-05-18 00:34:11,566 - Initializing
> ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-18 00:34:11,569
- RuntimeException thrown from GraphiteReporter#report. Exception was suppressed.
> java.util.ConcurrentModificationException: null
> 	at java.util.HashMap$HashIterator.nextNode( ~[na:1.8.0_91]
> 	at java.util.HashMap$ ~[na:1.8.0_91]
> 	at org.apache.cassandra.metrics.TableMetrics$33.getValue( ~[apache-cassandra-3.0.6.jar:3.0.6]
> 	at org.apache.cassandra.metrics.TableMetrics$33.getValue( ~[apache-cassandra-3.0.6.jar:3.0.6]
> 	at com.codahale.metrics.graphite.GraphiteReporter.reportGauge(
> 	at
> 	at ~[metrics-core-3.1.0.jar:3.1.0]
> 	at com.codahale.metrics.ScheduledReporter$ ~[metrics-core-3.1.0.jar:3.1.0]
> 	at java.util.concurrent.Executors$ [na:1.8.0_91]
> 	at java.util.concurrent.FutureTask.runAndReset( [na:1.8.0_91]
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker( [na:1.8.0_91]
> 	at java.util.concurrent.ThreadPoolExecutor$ [na:1.8.0_91]
> 	at [na:1.8.0_91]
> {code}

This message was sent by Atlassian JIRA

View raw message