incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Brosius <dbros...@mebigfatguy.com>
Subject Re: Recover from startup problems
Date Tue, 23 Aug 2011 03:35:37 GMT
0.8.4 seems to fail to load in the same way. When i deleted the data 
directory, it appears to start up correctly, I see

INFO 23:30:44,067 JNA not found. Native methods will be disabled.
  INFO 23:30:44,100 Loading settings from 
file:/home/dave/apache-cassandra-0.8.4/conf/cassandra.yaml
  INFO 23:30:44,387 DiskAccessMode 'auto' determined to be standard, 
indexAccessMode is standard
  INFO 23:30:44,710 Global memtable threshold is enabled at 245MB
  INFO 23:30:44,713 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,715 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,716 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,717 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,718 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,719 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:44,934 Creating new commitlog segment 
/var/lib/cassandra/commitlog/CommitLog-1314070244934.log
  INFO 23:30:44,994 Couldn't detect any schema definitions in local storage.
  INFO 23:30:45,001 Found table data in data directories. Consider using 
the CLI to define your schema.
  INFO 23:30:45,004 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,011 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,012 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,013 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,014 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,015 Removing compacted SSTable files (see 
http://wiki.apache.org/cassandra/MemtableSSTable)
  INFO 23:30:45,134 Replaying 
/var/lib/cassandra/commitlog/CommitLog-1313973548456.log
  INFO 23:30:45,239 Finished reading 
/var/lib/cassandra/commitlog/CommitLog-1313973548456.log
  INFO 23:30:45,244 Skipped 44 mutations from unknown (probably removed) 
CF with id 1001
  INFO 23:30:45,256 Enqueuing flush of 
Memtable-LocationInfo@18930675(186/232 serialized/live bytes, 5 ops)
  INFO 23:30:45,259 Writing Memtable-LocationInfo@18930675(186/232 
serialized/live bytes, 5 ops)
  INFO 23:30:45,274 Enqueuing flush of 
Memtable-Migrations@10140210(23897/29871 serialized/live bytes, 3 ops)
  INFO 23:30:45,275 Enqueuing flush of 
Memtable-Schema@3564915(11042/13802 serialized/live bytes, 12 ops)
  INFO 23:30:45,578 Completed flushing 
/var/lib/cassandra/data/system/LocationInfo-g-1-Data.db (296 bytes)
  INFO 23:30:45,602 Writing Memtable-Migrations@10140210(23897/29871 
serialized/live bytes, 3 ops)
  INFO 23:30:45,854 Completed flushing 
/var/lib/cassandra/data/system/Migrations-g-1-Data.db (23961 bytes)
  INFO 23:30:45,857 Writing Memtable-Schema@3564915(11042/13802 
serialized/live bytes, 12 ops)
  INFO 23:30:46,080 Completed flushing 
/var/lib/cassandra/data/system/Schema-g-1-Data.db (11274 bytes)
  INFO 23:30:46,095 Log replay complete, 12 replayed mutations
  INFO 23:30:46,166 Enqueuing flush of 
Memtable-LocationInfo@17711949(29/36 serialized/live bytes, 1 ops)
  INFO 23:30:46,167 Writing Memtable-LocationInfo@17711949(29/36 
serialized/live bytes, 1 ops)
  INFO 23:30:46,440 Completed flushing 
/var/lib/cassandra/data/system/LocationInfo-g-2-Data.db (80 bytes)
  INFO 23:30:47,205 Upgrading to 0.7. Purging hints if there are any. 
Old hints will be snapshotted.
  INFO 23:30:47,352 Cassandra version: 0.8.4
  INFO 23:30:47,353 Thrift API version: 19.10.0
  INFO 23:30:47,353 Loading persisted ring state
  INFO 23:30:47,355 Starting up server gossip
  INFO 23:30:47,360 Enqueuing flush of 
Memtable-LocationInfo@5604828(123/153 serialized/live bytes, 2 ops)
  INFO 23:30:47,361 Writing Memtable-LocationInfo@5604828(123/153 
serialized/live bytes, 2 ops)
  INFO 23:30:47,608 Completed flushing 
/var/lib/cassandra/data/system/LocationInfo-g-3-Data.db (231 bytes)
  INFO 23:30:47,658 Starting Messaging Service on localhost/127.0.0.1:7000
  INFO 23:30:47,689 Using saved token 
118080104480458226093045236055971412420
  INFO 23:30:47,691 Enqueuing flush of 
Memtable-LocationInfo@11872808(53/66 serialized/live bytes, 2 ops)
  INFO 23:30:47,692 Writing Memtable-LocationInfo@11872808(53/66 
serialized/live bytes, 2 ops)
  INFO 23:30:47,968 Completed flushing 
/var/lib/cassandra/data/system/LocationInfo-g-4-Data.db (163 bytes)
  INFO 23:30:47,974 Node localhost/127.0.0.1 state jump to normal
  INFO 23:30:47,978 Will not load MX4J, mx4j-tools.jar is not in the 
classpath


On 08/22/2011 11:24 PM, Jonathan Ellis wrote:
> It's erroring out trying to load the schema itself, though, which
> isn't supposed to happen.
>
> On Mon, Aug 22, 2011 at 10:14 PM, Dave Brosius<dbrosius@mebigfatguy.com>  wrote:
>> Sure i'll try that, but I'm pretty sure it was creating a column family
>> without any column meta data (types), then, client.insert'ing a ByteBuffer
>> that wasn't based on bytes from a String.getBytes call.
>>
>>
>>
>> On 08/22/2011 11:09 AM, Jonathan Ellis wrote:
>>> Yes, you can blow away both the data and commitlog directories and
>>> restart, but can you try these first to troubleshoot?
>>>
>>> 1. make a copy of the commitlog directory
>>> 2. downgrade to 0.8 with no other changes, to see if it's something on
>>> the new read path
>>> 2a. if 0.8 starts up then we will fix the read code in trunk
>>> 2b. if 0.8 doesn't start up either, remove the data directory but NOT
>>> commitlog, and restart.  this will cause commitlog to be replayed --
>>> with luck whatever is causing the problem is still in there, so if it
>>> breaks again, we have a reproducible case
>>>
>>> Thanks!
>>>
>>> On Mon, Aug 22, 2011 at 1:16 AM, Dave Brosius<dbrosius@mebigfatguy.com>
>>>   wrote:
>>>> Greetings, I'm running head from source, and now when i try to start up
>>>> the
>>>> database, i get the following exception which causes client connection
>>>> failures. I'm fine with blowing away the database, just playing, but
>>>> wanted
>>>> to know if there is a safe way to do this.
>>>>
>>>> Exception encountered during startup.
>>>> java.lang.RuntimeException: error reading 1 of 3
>>>>     at
>>>>
>>>> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
>>>>     at
>>>>
>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>>>>     at
>>>>
>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:107)
>>>>     at
>>>>
>>>> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:194)
>>>>     at
>>>> org.apache.cassandra.utils.MergeIterator.<init>(MergeIterator.java:47)
>>>>     at
>>>>
>>>> org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:142)
>>>>     at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:66)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:96)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:221)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1285)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1169)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1120)
>>>>     at
>>>> org.apache.cassandra.db.DefsTable.loadFromStorage(DefsTable.java:83)
>>>>     at
>>>>
>>>> org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:507)
>>>>     at
>>>>
>>>> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:159)
>>>>     at
>>>>
>>>> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:335)
>>>>     at
>>>> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
>>>> Caused by: java.nio.channels.ClosedChannelException
>>>>     at
>>>>
>>>> org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:268)
>>>>     at java.io.RandomAccessFile.readByte(RandomAccessFile.java:589)
>>>>     at
>>>>
>>>> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:356)
>>>>     at
>>>>
>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:367)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:82)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:72)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
>>>>     at
>>>>
>>>> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
>>>>
>>>
>>
>
>


Mime
View raw message