incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yulian Oifa <oifa.yul...@gmail.com>
Subject Cassandra Strange behaviour
Date Mon, 14 Apr 2014 16:22:43 GMT
Hello to all
I have cassandra cluster with 3 nodes and RF=3 writing with Quorum.
Application wrote today several millions of records to specific CF.
After that one of servers went wild , he eats up the disk.
As i see from logs hinted handoff from 2 other servers are occuring to disk
server.
On this server i see that data is flushed to disk each several seconds :
 WARN [CompactionExecutor:249] 2014-04-14 19:17:38,633
CompactionManager.java (line 509) insufficient space to compact all
requested files
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-548-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db')
 WARN [CompactionExecutor:249] 2014-04-14 19:17:58,647
CompactionManager.java (line 509) insufficient space to compact all
requested files
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-978-Data.db')
 INFO [COMMIT-LOG-WRITER] 2014-04-14 19:18:06,232 CommitLogSegment.java
(line 59) Creating new commitlog segment
/opt/cassandra/commitlog/CommitLog-1397492286232.log
 WARN [CompactionExecutor:249] 2014-04-14 19:18:18,648
CompactionManager.java (line 509) insufficient space to compact all
requested files
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1060-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-890-Data.db')
ERROR [CompactionExecutor:249] 2014-04-14 19:18:18,649
CompactionManager.java (line 513) insufficient space to compact even the
two smallest files, aborting
 INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 MeteredFlusher.java
(line 62) flushing high-traffic column family CFS(Keyspace='USER_DATA',
ColumnFamily='freeNumbers')
 INFO [NonPeriodicTasks:1] 2014-04-14 19:18:25,228 ColumnFamilyStore.java
(line 1128) Enqueuing flush of
Memtable-freeNumbers@1950635535(37693440/309874109
serialized/live bytes, 837632 ops)
 INFO [FlushWriter:22] 2014-04-14 19:18:25,229 Memtable.java (line 237)
Writing Memtable-freeNumbers@1950635535(37693440/309874109 serialized/live
bytes, 837632 ops)
 INFO [FlushWriter:22] 2014-04-14 19:18:26,871 Memtable.java (line 254)
Completed flushing /opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db
(38103944 bytes)
 INFO [CompactionExecutor:251] 2014-04-14 19:18:26,872
CompactionManager.java (line 542) Compacting Minor:
[SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1065-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1063-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1064-Data.db'),
SSTableReader(path='/opt/cassandra/data/USER_DATA/freeNumbers-g-1066-Data.db')]
 INFO [CompactionExecutor:251] 2014-04-14 19:18:26,878
CompactionController.java (line 146) Compacting large row
USER_DATA/freeNumbers:8bdf9678-6d70-11e3-85ab-80e385abf85d (151810145
bytes) incrementally


However total data of this CF is around 4.5 GB , while disk usage for this
CF on this server overcomes 20GB.
I have tried restart of this server , cyclic restart of all servers and no
luck it continues to write data .i can not run compact also....
How can i stop that?
Best regards
Yulian Oifa

Mime
View raw message