incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Cassandra Crash
Date Wed, 16 Mar 2011 03:10:34 GMT
I would scrub again w/ 0.7.4.

On Tue, Mar 15, 2011 at 6:38 PM, Sanjeev Kulkarni <sanjeev@locomatix.com> wrote:
> Hey Jonathan,
> Thanks for the reply.
> I was earlier running 0.7.2 and upgraded it to 0.7.3. Looks like I had to
> run the nodetool scrub command to sanitize the sstables because of the
> bloomfilter bug. I did that and the Assert error went away but I'm getting
> Java Heap Space Out of Memory error. I again upgraded to 0.7.4 which is just
> released but the OOM crash remains.
> The log is attached below. Row caching is disabled and key caching is set to
> default(200000). The max heap space that I'm giving is pretty large(6G). Do
> you think reducing the key caching will help?
> Thanks again!
> java.lang.OutOfMemoryError: Java heap space
>         at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:269)
>         at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
>         at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272)
>         at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>         at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>         at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180)
>         at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:130)
>         at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72)
>         at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:58)
>         at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1353)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>         at org.apache.cassandra.db.Table.getRow(Table.java:333)
>         at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60)
>         at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> On Tue, Mar 15, 2011 at 2:27 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Did you upgrade from an earlier version?  Did you read NEWS.txt?
>>
>> On Tue, Mar 15, 2011 at 4:21 PM, Sanjeev Kulkarni <sanjeev@locomatix.com>
>> wrote:
>> > Hey guys,
>> > Have started facing a crash in my cassandra while reading. Here are the
>> > details.
>> > 1. single node. replication factor of 1
>> > 2. Cassandra version 0.7.3
>> > 3. Single keyspace. 5 column families.
>> > 4. No super columns
>> > 5. My data model is a little bit skewed. It results in having several
>> > small
>> > rows and one really big row(lots of columns). cfstats say that on one
>> > column
>> > family the Compacted row maximum size is 5960319812. Not sure if this is
>> > the
>> > problem.
>> > 6. Starting cassandra has no issues. I give a max heap size of 6G.
>> > 7. I then start reading a bunch of rows including the really long row.
>> > After
>> > some point cassandra starts crashing.
>> > The log is
>> > ERROR [ReadStage:30] 2011-03-15 16:52:27,598
>> > AbstractCassandraDaemon.java
>> > (line 114) Fatal exception in thread Thread[ReadStage:30,5,main]
>> > java.lang.AssertionError
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180)
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:134)
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131)
>> >         at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >         at
>> >
>> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60)
>> >         at
>> >
>> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:662)
>> > ERROR [ReadStage:13] 2011-03-15 16:52:27,600
>> > AbstractCassandraDaemon.java
>> > (line 114) Fatal exception in thread Thread[ReadStage:13,5,main]
>> > java.lang.AssertionError
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:180)
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:134)
>> >         at
>> >
>> > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:72)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:59)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1311)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1131)
>> >         at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >         at
>> >
>> > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:60)
>> >         at
>> >
>> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:662)
>> > and several more of these errors
>> >
>> > Thanks for the help. Let me know if you need more info.
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message