cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yatong Zhang <bluefl...@gmail.com>
Subject Re: bugs report for 2.0.7
Date Wed, 30 Apr 2014 00:48:49 GMT
Here is another type of exception, seems all are I/O related:

 INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java (line
> 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6956
> (447252 bytes)
>  INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6958
> (257 bytes)
>  INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/compaction_history/system-compaction_history-jb-6957
> (257 bytes)
>  INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
> Initializing system.batchlog
>  INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
> Initializing system.sstable_activity
>  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084 (1562
> bytes)
>  INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083 (2075
> bytes)
>  INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
> (line 223) Opening
> /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085 (1555
> bytes)
>  INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
> reading saved cache
> /data1/saved_caches/system-sstable_activity-KeyCache-b.db
>  INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
> Initializing system.peer_events
>  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
> (line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-181
> (12342 bytes)
>  INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
> Initializing system.compactions_in_progress
>  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
> (line 223) Opening
> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-36448
> (167 bytes)
> ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
> (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException
>         at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:110)
>         at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
>         at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
>         at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:458)
>         at
> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422)
>         at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203)
>         at
> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184)
>         at
> org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.EOFException
>         at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:589)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
>         at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:85)
>         ... 12 more
>  INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
> Initializing system.hints
>  INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
> reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
>  INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
> Initializing system.schema_keyspaces
>



On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <blueflycn@gmail.com> wrote:

> I am pretty sure the disk has plenty of space, I am sure of that. I
> restarted cassandra and everything went fine again.
>
> It's really wired
>
>
> On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylvain@datastax.com>wrote:
>
>> The important part of that stack trace is "java.io.IOException: No space
>> left on device", your disks are full (and it's not really a bug that
>> Cassandra error out in that case).
>>
>> --
>> Sylvain
>>
>>
>> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <blueflycn@gmail.com>
>> wrote:
>>
>> > Hi there,
>> >
>> > Sorry if this is not the right place to report bugs. I am using 2.0.7
>> and I
>> > have a 10 boxes clusters with about 200TB capacity. I just found I had 3
>> > boxes with error exceptions. With datastax opscenter I can see these
>> three
>> > nodes lost connections (no reponse), but after I sshed to these server,
>> > cassandara were still running, and the 'system.log' still had logs.
>> >
>> > I think this might be a bug so any one would kindly help to investigate
>> > into it? Thanks~
>> >
>> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249
>> CassandraDaemon.java
>> > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
>> > > FSWriteError in
>> > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db
>> > >         at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
>> > >         at
>> > >
>> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> > >         at
>> > >
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
>> > >         at
>> > >
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> > >         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> > >         at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > >         at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > >         at java.lang.Thread.run(Thread.java:744)
>> > > Caused by: java.io.IOException: No space left on device
>> > >         at java.io.FileOutputStream.write(Native Method)
>> > >         at java.io.FileOutputStream.write(FileOutputStream.java:295)
>> > >         at
>> java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
>> > >         at
>> > >
>> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
>> > >         at
>> > >
>> >
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)
>> > >         ... 13 more
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406
>> StorageService.java
>> > > (line 367) Stopping gossiper
>> > >  WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406
>> StorageService.java
>> > > (line 281) Stopping gossip by operator request
>> > >  INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java
>> (line
>> > > 1271) Announcing shutdown
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406
>> StorageService.java
>> > > (line 372) Stopping RPC server
>> > >  INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
>> > > (line 141) Stop listening to thrift clients
>> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417
>> StorageService.java
>> > > (line 377) Stopping native transport
>> > >  INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
>> > > 181) Stop listening for CQL clients
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message