incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: Exceptions on 0.7.0
Date Thu, 10 Feb 2011 00:42:05 GMT
Out of curiosity, do you really have on the order of 1,986,622,313 elements
(I believe elements=keys) in the cf?

 

Dan

 

From: shimi [mailto:shimi.k@gmail.com] 
Sent: February-09-11 15:06
To: user@cassandra.apache.org
Subject: Exceptions on 0.7.0

 

I have a 4 node test cluster were I test the port to 0.7.0 from 0.6.X

On 3 out of the 4 nodes I get exceptions in the log.

I am using RP.

Changes that I did:

1. changed the replication factor from 3 to 4

2. configured the nodes to use Dynamic Snitch

3. RR of 0.33

 

I run repair on 2 nodes  before I noticed the errors. One of them is having
the first error and the other the second.

I restart the nodes but I still get the exceptions.

 

The following Exception I get from 2 nodes:

 WARN [CompactionExecutor:1] 2011-02-09 19:50:51,281 BloomFilter.java (line
84) Cannot provide an optimal Bloom

Filter for 1986622313 elements (1/4 buckets per element).

ERROR [CompactionExecutor:1] 2011-02-09 19:51:10,190
AbstractCassandraDaemon.java (line 91) Fatal exception in 

thread Thread[CompactionExecutor:1,1,main]

java.io.IOError: java.io.EOFException

        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentity
Iterator.java:105)

        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentity
Iterator.java:34)

        at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIter
ator.java:284)

        at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIt
erator.java:326)

        at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIte
rator.java:230)

        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav
a:68)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)

        at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)

        at
org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:7
6)

        at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)

        at
org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88
)

        at
org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterato
r.java:136)

        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:107)

        at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.jav
a:42)

        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav
a:73)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)

        at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(Filter
Iterator.java:183)

        at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterat
or.java:94)

        at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.jav
a:323)

        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)

        at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)

        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.EOFException

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

        at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)

        at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:7
6)

        at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:3
5)

        at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentity
Iterator.java:101)

        ... 29 more

 

 

On another node I get:

 

ERROR [pool-1-thread-2] 2011-02-09 19:48:32,137 Cassandra.java (line 2876)
Internal error processing get_range_

slices

java.lang.RuntimeException: error reading 1 of 1970563183

        at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleS
liceReader.java:82)

        at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleS
liceReader.java:39)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)

        at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableS
liceIterator.java:108)

        at
org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(Collat
ingIterator.java:364)

        at
org.apache.commons.collections.iterators.CollatingIterator.hasNext(Collating
Iterator.java:217)

        at
org.apache.cassandra.db.RowIteratorFactory$3.getReduced(RowIteratorFactory.j
ava:136)

        at
org.apache.cassandra.db.RowIteratorFactory$3.getReduced(RowIteratorFactory.j
ava:106)

        at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.jav
a:73)

        at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator
.java:136)

        at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131
)

        at org.apache.cassandra.db.RowIterator.hasNext(RowIterator.java:49)

        at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.ja
va:1294)

        at
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:43
8)

        at
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer
.java:473)

        at
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cas
sandra.java:2868)

        at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)

        at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(Custom
TThreadPoolServer.java:1

67)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.EOFException

        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

        at
org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)

        at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:9
4)

        at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:3
5)

        at
org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleS
liceReader.java:78)

        ... 21 more

 

any idea what went wrong?

Shimi

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus Database: 271.1.1/3432 - Release Date: 02/09/11
02:34:00


Mime
View raw message