incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nury Redjepow <nreje...@mail.ru>
Subject CorruptedBlockException
Date Thu, 05 Jul 2012 12:04:51 GMT
Hello to all,

 I have cassandra instance I'm trying to use to store millions of file with size ~ 3MB. Data
structure is simple, 1 row for 1 file, with row key being the id of file.
I'm loaded 1GB of data, and total available space is 10GB. And after a few hour, all the
available space was taken. In log, it says that while compaction a CorruptedBlockException
has occured. But I don't understand, how was all the available space taken away.

Data structure
CREATE KEYSPACE largeobjectsWITH placement_strategy = 'SimpleStrategy'
AND strategy_options={replication_factor:1};

create column family content
  with column_type = 'Standard'  
  and comparator = 'UTF8Type'
  and default_validation_class = 'BytesType'
  and key_validation_class = 'TimeUUIDType'
  and read_repair_chance = 0.1
  and dclocal_read_repair_chance = 0.0
  and gc_grace = 864000
  and min_compaction_threshold = 4
  and max_compaction_threshold = 32
  and replicate_on_write = true
  and compaction_strategy = 'SizeTieredCompactionStrategy'
  and caching = 'keys_only';


Log messages

INFO [FlushWriter:9] 2012-07-04 19:56:00,783 Memtable.java (line 266) Writing Memtable-content@240294142(3955135/49439187
serialized/live bytes, 91 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:00,814 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1608-Data.db (1991862 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=78253718)
 INFO [OptionalTasks:1] 2012-07-04 19:56:02,784 MeteredFlusher.java (line 62) flushing high-traffic
column family CFS(Keyspace='largeobjects', ColumnFamily='
content') (estimated 46971537 bytes)
 INFO [OptionalTasks:1] 2012-07-04 19:56:02,784 ColumnFamilyStore.java (line 633) Enqueuing
flush of Memtable-content@1755783901(3757723/46971537 serialized/
live bytes, 121 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:02,785 Memtable.java (line 266) Writing Memtable-content@1755783901(3757723/46971537
serialized/live bytes, 121 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:02,835 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1609-Data.db (1894897 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=82028986)
 INFO [OptionalTasks:1] 2012-07-04 19:56:04,785 MeteredFlusher.java (line 62) flushing high-traffic
column family CFS(Keyspace='largeobjects', ColumnFamily='
content') (estimated 56971025 bytes)
 INFO [OptionalTasks:1] 2012-07-04 19:56:04,785 ColumnFamilyStore.java (line 633) Enqueuing
flush of Memtable-content@1441175031(4557682/56971025 serialized/
live bytes, 124 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:04,786 Memtable.java (line 266) Writing Memtable-content@1441175031(4557682/56971025
serialized/live bytes, 124 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:04,814 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1610-Data.db (2287280 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=86604648)
 INFO [CompactionExecutor:39] 2012-07-04 19:56:04,815 CompactionTask.java (line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/largeobjects/con
tent/largeobjects-content-hd-1610-Data.db'), SSTableReader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1608-Data.db'),
SSTable
Reader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1609-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/largeobjects/co
ntent/largeobjects-content-hd-1607-Data.db')]
 INFO [OptionalTasks:1] 2012-07-04 19:56:05,786 MeteredFlusher.java (line 62) flushing high-traffic
column family CFS(Keyspace='largeobjects', ColumnFamily='
content') (estimated 28300225 bytes)
 INFO [OptionalTasks:1] 2012-07-04 19:56:05,786 ColumnFamilyStore.java (line 633) Enqueuing
flush of Memtable-content@1828084851(2264018/28300225 serialized/
live bytes, 38 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:05,787 Memtable.java (line 266) Writing Memtable-content@1828084851(2264018/28300225
serialized/live bytes, 38 ops)
 INFO [FlushWriter:9] 2012-07-04 19:56:05,823 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1612-Data.db (1134604 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=88874176)
ERROR [CompactionExecutor:39] 2012-07-04 19:56:06,667 AbstractCassandraDaemon.java (line 134)
Exception in thread Thread[CompactionExecutor:39,1,main]
java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1610-Data.db
): corruption detected, chunk at 1573104 of length 65545.
 at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
 at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:99)
 at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:164)
 at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
 at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
 at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
 at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
 at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
 at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
 at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1610-Data.db):
cor
ruption detected, chunk at 1573104 of length 65545.
 at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98)
 at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
 at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
 at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
 at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
 at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95)
 at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
 at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119)
 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
 at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
 at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:112)
 ... 20 more
 INFO [OptionalTasks:1] 2012-07-04 19:57:00,796 MeteredFlusher.java (line 62) flushing high-traffic
column family CFS(Keyspace='largeobjects', ColumnFamily='
content') (estimated 38711275 bytes)
 INFO [OptionalTasks:1] 2012-07-04 19:57:00,796 ColumnFamilyStore.java (line 633) Enqueuing
flush of Memtable-content@1363920595(3096902/38711275 serialized/
live bytes, 74 ops)
 INFO [FlushWriter:9] 2012-07-04 19:57:00,796 Memtable.java (line 266) Writing Memtable-content@1363920595(3096902/38711275
serialized/live bytes, 74 ops)
 INFO [FlushWriter:9] 2012-07-04 19:57:00,821 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1613-Data.db (1553451 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=91981808)
 INFO [CompactionExecutor:40] 2012-07-04 19:57:00,822 CompactionTask.java (line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/largeobjects/con
tent/largeobjects-content-hd-1610-Data.db'), SSTableReader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1613-Data.db'),
SSTable
Reader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1608-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/largeobjects/co
ntent/largeobjects-content-hd-1609-Data.db'), SSTableReader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1607-Data.db'),
SSTabl
eReader(path='/var/lib/cassandra/data/largeobjects/content/largeobjects-content-hd-1612-Data.db')]
 INFO [OptionalTasks:1] 2012-07-04 19:57:01,797 MeteredFlusher.java (line 62) flushing high-traffic
column family CFS(Keyspace='largeobjects', ColumnFamily='
content') (estimated 27750950 bytes)
 INFO [OptionalTasks:1] 2012-07-04 19:57:01,797 ColumnFamilyStore.java (line 633) Enqueuing
flush of Memtable-content@289600485(2220076/27750950 serialized/l
ive bytes, 70 ops)
 INFO [FlushWriter:9] 2012-07-04 19:57:01,797 Memtable.java (line 266) Writing Memtable-content@289600485(2220076/27750950
serialized/live bytes, 70 ops)
 INFO [FlushWriter:9] 2012-07-04 19:57:01,819 Memtable.java (line 307) Completed flushing
/var/lib/cassandra/data/largeobjects/content/largeobjects-content-h
d-1615-Data.db (1114538 bytes) for commitlog position ReplayPosition(segmentId=24245436475633,
position=94212034)
ERROR [ReadStage:263] 2012-07-04 19:57:02,599 AbstractCassandraDaemon.java (line 134) Exception
in thread Thread[ReadStage:263,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: error reading 1 of 1
 at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1254)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: error reading 1 of 1
 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39)
 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
 at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
 at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
 at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:90)
 at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:47)
 at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:137)
 at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:283)
 at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1321)
 
 
Mime
View raw message