cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Daubman (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5605) Crash caused by insufficient disk space to flush
Date Wed, 07 Aug 2013 15:56:51 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732098#comment-13732098
] 

Aaron Daubman edited comment on CASSANDRA-5605 at 8/7/13 3:56 PM:
------------------------------------------------------------------

Apologies if this isn't directly relevant, but I seem to be experiencing the same issue using
1.2.8 launched via ccm for integration testing. One differentiating feature here is that this
happened the first time the node was ever brought up (0 data). The integration tests attempting
to use the ccm cluster never completed due to this hang, but all they do is test creation
of a fairly simple schema and then attempt to write and then read back a single row. There
is plenty of free disk space available...

Here's what was in the log:
 INFO [main] 2013-08-07 14:56:46,763 CassandraDaemon.java (line 118) Logging initialized
 INFO [main] 2013-08-07 14:56:46,807 CassandraDaemon.java (line 145) JVM vendor/version: Java
HotSpot(TM) 64-Bit Server VM/1.7.0_25
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 183) Heap size: 8248098816/8248098816
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 184) Classpath: /opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/main:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/thrift:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/antlr-3.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-sources-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-cli-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-codec-1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-lang-2.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/compress-lzf-0.8.4.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/concurrentlinkedhashmap-lru-1.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/guava-13.0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/high-scale-lib-1.1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-core-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-mapper-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jbcrypt-0.3m.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jline-1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/json-simple-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/libthrift-0.7.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/log4j-1.2.16.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/lz4-1.1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/metrics-core-2.0.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/netty-3.5.9.Final.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/servlet-api-2.5-20081211.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-api-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-log4j12-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snakeyaml-1.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snappy-java-1.0.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snaptree-0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar
 INFO [main] 2013-08-07 14:56:46,822 CLibrary.java (line 65) JNA not found. Native methods
will be disabled.
 INFO [main] 2013-08-07 14:56:46,891 DatabaseDescriptor.java (line 132) Loading settings from
file:/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf/cassandra.yaml
 INFO [main] 2013-08-07 14:56:47,821 DatabaseDescriptor.java (line 150) Data files directories:
[/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/data]
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 151) Commit log directory:
/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/commitlogs
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 191) DiskAccessMode 'auto'
determined to be mmap, indexAccessMode is mmap
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 205) disk_failure_policy
is stop
 INFO [main] 2013-08-07 14:56:48,000 DatabaseDescriptor.java (line 273) Global memtable threshold
is enabled at 2622MB
 INFO [main] 2013-08-07 14:56:49,142 DatabaseDescriptor.java (line 401) Not using multi-threaded
compaction
 INFO [main] 2013-08-07 14:56:52,610 CacheService.java (line 111) Initializing key cache with
capacity of 100 MBs.
 INFO [main] 2013-08-07 14:56:52,675 CacheService.java (line 140) Scheduling key cache save
to each 14400 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:52,678 CacheService.java (line 154) Initializing row cache with
capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO [main] 2013-08-07 14:56:52,698 CacheService.java (line 166) Scheduling row cache save
to each 0 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:57,163 DatabaseDescriptor.java (line 535) Couldn't detect any
schema definitions in local storage.
 INFO [main] 2013-08-07 14:56:57,164 DatabaseDescriptor.java (line 540) To create keyspaces
and column families, see 'help create' in cqlsh.
 INFO [main] 2013-08-07 14:56:57,258 CommitLog.java (line 120) No commitlog files found; skipping
replay
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 456) Cassandra version: 1.2.8-SNAPSHOT
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 457) Thrift API version: 19.36.0
 INFO [main] 2013-08-07 14:56:57,716 StorageService.java (line 458) CQL supported versions:
2.0.0,3.0.5 (default: 3.0.5)
 INFO [main] 2013-08-07 14:56:57,795 StorageService.java (line 483) Loading persisted ring
state
 INFO [main] 2013-08-07 14:56:57,805 StorageService.java (line 564) Starting up server gossip
 WARN [main] 2013-08-07 14:56:57,823 SystemTable.java (line 573) No host ID found, created
6abf69c6-8472-4607-96b7-4a532ed8e2a8 (Note: This should happen exactly once per node).
 INFO [main] 2013-08-07 14:56:57,852 ColumnFamilyStore.java (line 630) Enqueuing flush of
Memtable-local@2038679449(367/367 serialized/live bytes, 15 ops)
ERROR [FlushWriter:1] 2013-08-07 14:56:57,860 CassandraDaemon.java (line 192) Exception in
thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 452 bytes
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:42)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,243 GCInspector.java (line 119) GC for ParNew:
2786 ms for 1 collections, 230275096 used; max is 8248098816
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,250 StatusLogger.java (line 53) Pool Name   
                Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,291 StatusLogger.java (line 68) MemtablePostFlusher
              1         1         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,292 StatusLogger.java (line 68) FlushWriter 
                     0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,293 StatusLogger.java (line 68) commitlog_archiver
               0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,294 StatusLogger.java (line 73) CompactionManager
                0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,522 StatusLogger.java (line 85) MessagingService
               n/a       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,523 StatusLogger.java (line 95) Cache Type  
                  Size                 Capacity               KeysToSave                 
                                       Provider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,524 StatusLogger.java (line 96) KeyCache    
                     0                104857600                      all                 
                                               
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 102) RowCache   
                      0                        0                      all              org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 109) ColumnFamily
               Memtable ops,data
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) system.local
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) system.peers
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) system.batchlog
                          0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) system.NodeIdInfo
                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) system.LocationInfo
                      0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) system.Schema
                            0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) system.Migrations
                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) system.schema_keyspaces
                8,251
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.schema_columns
              398,24717
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.schema_columnfamilies
          369,22187
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.IndexInfo
                         0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) system.range_xfers
                       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) system.peer_events
                       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system.hints
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system.HintsColumnFamily
                 0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system_traces.sessions
                   0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,533 StatusLogger.java (line 112) system_traces.events
                     0,0

                
      was (Author: daubman):
    Apologies if this isn't directly relevant, but I seem to be experiencing the same issue
using 1.2.8 launched via ccm for integration testing. One differentiating feature here is
that this happened the first time the node was ever brought up (0 data). The integration tests
attempting to use the ccm cluster never completed due to this hang, but all they do is test
creation of a fairly simple schema and then attempt to write and then read back a single row.

Here's what was in the log:
 INFO [main] 2013-08-07 14:56:46,763 CassandraDaemon.java (line 118) Logging initialized
 INFO [main] 2013-08-07 14:56:46,807 CassandraDaemon.java (line 145) JVM vendor/version: Java
HotSpot(TM) 64-Bit Server VM/1.7.0_25
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 183) Heap size: 8248098816/8248098816
 INFO [main] 2013-08-07 14:56:46,808 CassandraDaemon.java (line 184) Classpath: /opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/main:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/build/classes/thrift:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/antlr-3.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/avro-1.4.0-sources-fixes.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-cli-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-codec-1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/commons-lang-2.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/compress-lzf-0.8.4.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/concurrentlinkedhashmap-lru-1.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/guava-13.0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/high-scale-lib-1.1.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-core-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jackson-mapper-asl-1.9.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jbcrypt-0.3m.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jline-1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/json-simple-1.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/libthrift-0.7.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/log4j-1.2.16.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/lz4-1.1.0.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/metrics-core-2.0.3.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/netty-3.5.9.Final.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/servlet-api-2.5-20081211.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-api-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/slf4j-log4j12-1.7.2.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snakeyaml-1.6.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snappy-java-1.0.5.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/snaptree-0.1.jar:/opt/ccmlib_cassandra/apache-cassandra-1.2.8-src/lib/jamm-0.2.5.jar
 INFO [main] 2013-08-07 14:56:46,822 CLibrary.java (line 65) JNA not found. Native methods
will be disabled.
 INFO [main] 2013-08-07 14:56:46,891 DatabaseDescriptor.java (line 132) Loading settings from
file:/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/conf/cassandra.yaml
 INFO [main] 2013-08-07 14:56:47,821 DatabaseDescriptor.java (line 150) Data files directories:
[/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/data]
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 151) Commit log directory:
/opt/ccmlib_cassandra/ccm/lwcdbng_test_cluster/node1/commitlogs
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 191) DiskAccessMode 'auto'
determined to be mmap, indexAccessMode is mmap
 INFO [main] 2013-08-07 14:56:47,822 DatabaseDescriptor.java (line 205) disk_failure_policy
is stop
 INFO [main] 2013-08-07 14:56:48,000 DatabaseDescriptor.java (line 273) Global memtable threshold
is enabled at 2622MB
 INFO [main] 2013-08-07 14:56:49,142 DatabaseDescriptor.java (line 401) Not using multi-threaded
compaction
 INFO [main] 2013-08-07 14:56:52,610 CacheService.java (line 111) Initializing key cache with
capacity of 100 MBs.
 INFO [main] 2013-08-07 14:56:52,675 CacheService.java (line 140) Scheduling key cache save
to each 14400 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:52,678 CacheService.java (line 154) Initializing row cache with
capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO [main] 2013-08-07 14:56:52,698 CacheService.java (line 166) Scheduling row cache save
to each 0 seconds (going to save all keys).
 INFO [main] 2013-08-07 14:56:57,163 DatabaseDescriptor.java (line 535) Couldn't detect any
schema definitions in local storage.
 INFO [main] 2013-08-07 14:56:57,164 DatabaseDescriptor.java (line 540) To create keyspaces
and column families, see 'help create' in cqlsh.
 INFO [main] 2013-08-07 14:56:57,258 CommitLog.java (line 120) No commitlog files found; skipping
replay
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 456) Cassandra version: 1.2.8-SNAPSHOT
 INFO [main] 2013-08-07 14:56:57,715 StorageService.java (line 457) Thrift API version: 19.36.0
 INFO [main] 2013-08-07 14:56:57,716 StorageService.java (line 458) CQL supported versions:
2.0.0,3.0.5 (default: 3.0.5)
 INFO [main] 2013-08-07 14:56:57,795 StorageService.java (line 483) Loading persisted ring
state
 INFO [main] 2013-08-07 14:56:57,805 StorageService.java (line 564) Starting up server gossip
 WARN [main] 2013-08-07 14:56:57,823 SystemTable.java (line 573) No host ID found, created
6abf69c6-8472-4607-96b7-4a532ed8e2a8 (Note: This should happen exactly once per node).
 INFO [main] 2013-08-07 14:56:57,852 ColumnFamilyStore.java (line 630) Enqueuing flush of
Memtable-local@2038679449(367/367 serialized/live bytes, 15 ops)
ERROR [FlushWriter:1] 2013-08-07 14:56:57,860 CassandraDaemon.java (line 192) Exception in
thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 452 bytes
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:42)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,243 GCInspector.java (line 119) GC for ParNew:
2786 ms for 1 collections, 230275096 used; max is 8248098816
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,250 StatusLogger.java (line 53) Pool Name   
                Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,291 StatusLogger.java (line 68) MemtablePostFlusher
              1         1         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,292 StatusLogger.java (line 68) FlushWriter 
                     0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,293 StatusLogger.java (line 68) commitlog_archiver
               0         0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,294 StatusLogger.java (line 73) CompactionManager
                0         0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,522 StatusLogger.java (line 85) MessagingService
               n/a       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,523 StatusLogger.java (line 95) Cache Type  
                  Size                 Capacity               KeysToSave                 
                                       Provider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,524 StatusLogger.java (line 96) KeyCache    
                     0                104857600                      all                 
                                               
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 102) RowCache   
                      0                        0                      all              org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,525 StatusLogger.java (line 109) ColumnFamily
               Memtable ops,data
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) system.local
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,526 StatusLogger.java (line 112) system.peers
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) system.batchlog
                          0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,527 StatusLogger.java (line 112) system.NodeIdInfo
                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) system.LocationInfo
                      0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,528 StatusLogger.java (line 112) system.Schema
                            0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) system.Migrations
                        0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,529 StatusLogger.java (line 112) system.schema_keyspaces
                8,251
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.schema_columns
              398,24717
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.schema_columnfamilies
          369,22187
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,530 StatusLogger.java (line 112) system.IndexInfo
                         0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) system.range_xfers
                       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,531 StatusLogger.java (line 112) system.peer_events
                       0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system.hints
                             0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system.HintsColumnFamily
                 0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,532 StatusLogger.java (line 112) system_traces.sessions
                   0,0
 INFO [ScheduledTasks:1] 2013-08-07 15:14:08,533 StatusLogger.java (line 112) system_traces.events
                     0,0

                  
> Crash caused by insufficient disk space to flush
> ------------------------------------------------
>
>                 Key: CASSANDRA-5605
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5605
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.5
>         Environment: java version "1.7.0_15"
>            Reporter: Dan Hendry
>            Priority: Minor
>             Fix For: 2.0.1
>
>
> A few times now I have seen our Cassandra nodes crash by running themselves out of memory.
It starts with the following exception:
> {noformat}
> ERROR [FlushWriter:13000] 2013-05-31 11:32:02,350 CassandraDaemon.java (line 164) Exception
in thread Thread[FlushWriter:13000,5,main]
> java.lang.RuntimeException: Insufficient disk space to write 8042730 bytes
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:42)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:722)
> {noformat} 
> After which, it seems the MemtablePostFlusher stage gets stuck and no further memtables
get flushed: 
> {noformat} 
> INFO [ScheduledTasks:1] 2013-05-31 11:59:12,467 StatusLogger.java (line 68) MemtablePostFlusher
              1        32         0
> INFO [ScheduledTasks:1] 2013-05-31 11:59:12,469 StatusLogger.java (line 73) CompactionManager
                1         2
> {noformat} 
> What makes this ridiculous is that, at the time, the data directory on this node had
981GB free disk space (as reported by du). We primarily use STCS and at the time the aforementioned
exception occurred, at least one compaction task was executing which could have easily involved
981GB (or more) worth of input SSTables. Correct me if I am wrong but but Cassandra counts
data currently being compacted against available disk space. In our case, this is a significant
overestimation of the space required by compaction since a large portion of the data being
compacted has expired or is an overwrite.
> More to the point though, Cassandra should not crash because its out of disk space unless
its really actually out of disk space (ie, dont consider 'phantom' compaction disk usage when
flushing). I have seen one of our nodes die in this way before our alerts for disk space even
went off.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message