cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2618) DynamicSnitch race in adding latencies
Date Fri, 06 May 2011 22:38:03 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Brandon Williams updated CASSANDRA-2618:
----------------------------------------

    Attachment: 2618.txt

Patch to avoid the race between add() and clear().

> DynamicSnitch race in adding latencies
> --------------------------------------
>
>                 Key: CASSANDRA-2618
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2618
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Brandon Williams
>            Assignee: Brandon Williams
>         Attachments: 2618.txt
>
>
> ERROR 15:33:48,614 Fatal exception in thread Thread[ReadStage:264,5,main]
> java.lang.RuntimeException: java.util.NoSuchElementException
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:680)
> Caused by: java.util.NoSuchElementException
> 	at java.util.concurrent.LinkedBlockingDeque.removeFirst(LinkedBlockingDeque.java:401)
> 	at java.util.concurrent.LinkedBlockingDeque.remove(LinkedBlockingDeque.java:621)
> 	at org.apache.cassandra.locator.AdaptiveLatencyTracker.add(DynamicEndpointSnitch.java:288)
> 	at org.apache.cassandra.locator.DynamicEndpointSnitch.receiveTiming(DynamicEndpointSnitch.java:202)
> 	at org.apache.cassandra.net.MessagingService.addLatency(MessagingService.java:152)
> 	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> ERROR 15:33:48,615 Fatal exception in thread Thread[ReadStage:264,5,main]
> java.lang.RuntimeException: java.util.NoSuchElementException
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:680)
> Caused by: java.util.NoSuchElementException
> 	at java.util.concurrent.LinkedBlockingDeque.removeFirst(LinkedBlockingDeque.java:401)
> 	at java.util.concurrent.LinkedBlockingDeque.remove(LinkedBlockingDeque.java:621)
> 	at org.apache.cassandra.locator.AdaptiveLatencyTracker.add(DynamicEndpointSnitch.java:288)
> 	at org.apache.cassandra.locator.DynamicEndpointSnitch.receiveTiming(DynamicEndpointSnitch.java:202)
> 	at org.apache.cassandra.net.MessagingService.addLatency(MessagingService.java:152)
> 	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> What is happening that AdaptiveLatencyTracker.add is trying to add a latency, but the
deque is full, so it makes a second effort to remove an entry from the deque and then try
to add again.  However, when it tries to remove, the deque has already been emptied by DES.reset
call clear() on all the ALTs.  This bug has existed for a long time, but it's very rare and
difficult to trigger.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message