cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryon Spahn <bsp...@kitedesk.com>
Subject RE: Compaction causing listeners to stall
Date Tue, 08 Jul 2014 12:31:43 GMT
Robert,



New development today:



FSReadError in
/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db

        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:95)

        at
org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41)

        at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:323)

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)

        at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)

        at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)

        at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)

        at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)

        at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)

        at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)

        at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)

        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:185)

        at
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:271)

        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

        at java.lang.Thread.run(Thread.java:744)

Caused by: java.nio.channels.ClosedChannelException

        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)

        at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:250)

        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:101)

        at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87)

        ... 17 more



Followed by:



ERROR [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 364) Stopping gossiper

WARN [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 278) Stopping gossip by
operator request

INFO [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 Gossiper.java (line 1251) Announcing shutdown



After this the listeners are no longer available but the DB does not
officially die just hangs and needs a restart.



*From:* Robert Coli [mailto:rcoli@eventbrite.com]
*Sent:* Monday, July 7, 2014 6:55 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Compaction causing listeners to stall



On Mon, Jul 7, 2014 at 5:20 AM, Bryon Spahn <bspahn@kitedesk.com> wrote:

I am experiencing a strange issue where we run a compaction job weekly and
as a result, the listeners stall. This is a single node cluster running on
an i2.2xl instance in AWS. We are getting the message:



There are almost no cases where it makes sense to run a single node of
Cassandra, especially in production.



*[StorageServiceShutdownHook]*



I bet you a donut that you're OOMing the JVM. Stop doing that, and your
Cassandra node will stop crashing.



https://issues.apache.org/jira/browse/CASSANDRA-7507



Is probably the case you have just hit.



Basically, in some pathological circumstances, the JVM will send Cassandra
a signal that it handles as if you were an operator attempting a clean
shutdown. This probably usually does not succeed, but may be worth a shot.



=Rob

Mime
View raw message