cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arya Goudarzi (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5266) FileNotFoundException during AntiEntropy Stream Session after compaction
Date Mon, 10 Feb 2014 21:25:21 GMT


Arya Goudarzi commented on CASSANDRA-5266:

This weekend this issue was reproduced on our production cluster which runs 1.2.14 now. But,
we don't use secondary indexes this time. We lost a node a few days ago which I tried to replace.
Over the course of repair, lots of these FileNotFound exceptions showed up during streaming
sessions from other nodes to the node being repaired. As a result, the node being repaired
was hung on its repair task and had to be restarted. 

> FileNotFoundException during AntiEntropy Stream Session after compaction
> ------------------------------------------------------------------------
>                 Key: CASSANDRA-5266
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.6
>         Environment: Ubuntu 10.04
> Cassandra 1.1.6
> Jave 6 (64 Bit Sun JVM)
>            Reporter: Arya Goudarzi
>            Assignee: Yuki Morishita
> It seems that there is a race condition in the compaction process vs AntiEntropy Streams.
We often have repairs tossing lots of exceptions in the logs as follows:
> java.lang.RuntimeException: /var/lib/cassandra/data/keyspace_production/User/keyspace_production-User-hf-20056-Data.db
(No such file or directory)
>         at org.apache.cassandra.utils.FBUtilities.unchecked(
>         at
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> Caused by: /var/lib/cassandra/data/keyspace_production/User/keyspace_production-User-hf-20056-Data.db
(No such file or directory)
>         at Method)
>         at<init>(
>         at<init>(
>         at<init>(
>         at
>         at
>         at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(
>         at
>         ... 3 more
> Now about 1-2 minutes before that, I see Secondary Index buildup and compaction which
does compact the same SSTable to a new one. 
> INFO [Thread-90838] 2013-02-07 13:51:58,362 (line 126) Submitting
index build of publisher_id, for data in SSTableReader(path='/var/lib/cassandra/data/keyspace_production/User/keyspace_production-User-hf-20054-Data.db'),
>  INFO [CompactionExecutor:436] 2013-02-07 13:51:58,363 (line 109)
Compacting [SSTableReader(path='/var/lib/cassandra/data/keyspace_production/User/keyspace_production-User-hf-20055-Data.db'),
> Now, does this make the repair sessions to fail? It seems the node was trying to stream
a file to a node being repaired but apparently it did not.

This message was sent by Atlassian JIRA

View raw message