incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Siemens <rsiem...@greatergood.com>
Subject Titan/Cassandra Keyspace simply disappears
Date Fri, 08 Feb 2013 18:32:21 GMT

Just had something odd happen - my keyspace disappeared.  I'm curious if anyone has seen the
same or may have an explanation.

I am using Titan 0.2.0 and Cassandra 1.1.6.  I ran my application which creates a graph and
progressively builds it for a couple of hours.  But before completing, Cassandra seems to
simply lose track of the keyspace, "Recommender".  The exception is that it doesn't exist.
 So I check in cassandra-cli, and it also doesn't show the keyspace.  The data directory and
column family directories are there, but are empty.  Restarting cassandra doesn't change anything.

Ron


Here is part of my application log and the Cassandra log from the relevant time:

2013-02-07 17:58:02,012 [pool-10-thread-26] INFO  publisher.GraphPublisher - RelationProcessor
started fo
r items [47070, 34334, 34334, 34334, 42297, 34334, 34334]
2013-02-07 17:58:02,550 [pool-10-thread-36] ERROR publisher.GraphPublisher - Unhandled error
for account 
7314766 order 17489735
com.thinkaurelius.titan.core.TitanException: Could not read from storage after 3 attempts
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.readException(StandardTitanGraph.j
ava:162)
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:607)
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:592)
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.queryForEntries(StandardTitanGraph
.java:557)
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.loadRelations(StandardTitanGraph.j
ava:322)
        at com.thinkaurelius.titan.graphdb.transaction.StandardPersistTitanTx.loadRelations(StandardPersi
stTitanTx.java:155)
        at com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.ensureLoadedEdges(AbstractTitanVe
rtex.java:88)
        at com.thinkaurelius.titan.graphdb.vertices.StandardTitanVertex.getRelations(StandardTitanVertex.
java:68)
        at com.thinkaurelius.titan.graphdb.query.SimpleAtomicQuery.propertyIterator(SimpleAtomicQuery.jav
a:461)
        at com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.getProperty(AbstractTitanVertex.j
ava:115)
        at com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.getProperty(AbstractTitanVertex.j
ava:127)
        at com.charityusa.publisher.GraphPublisher.updateRelation(GraphPublisher.java:206)
        at com.charityusa.publisher.GraphPublisher.access$100(GraphPublisher.java:55)
        at com.charityusa.publisher.GraphPublisher$OrderProcessor.call(GraphPublisher.java:320)
        at com.charityusa.publisher.GraphPublisher$OrderProcessor.call(GraphPublisher.java:266)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure
in storage ba
ckend
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getS
lice(AstyanaxOrderedKeyColumnValueStore.java:148)
        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferedKeyColumnValueStore.getSlice(Buffer
edKeyColumnValueStore.java:36)
        at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.getSlice(Cons
istentKeyLockStore.java:82)
        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:602)
        ... 18 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException:
[host
=cirrus1(10.1.5.44):9160, latency=0(0), attempts=1] InvalidRequestException(why:Keyspace Recommender
does
 not exist)
        at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159
)
        at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImp
l.java:196)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImp
l.java:188)
        at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$1.execute(ThriftSyncConnectionFact
oryImpl.java:131)
        at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(Abstract
ExecuteWithFailoverImpl.java:52)
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailov
er(AbstractHostPartitionConnectionPool.java:229)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.
java:186)
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getS
lice(AstyanaxOrderedKeyColumnValueStore.java:145)
        ... 21 more
Caused by: InvalidRequestException(why:Keyspace Recommender does not exist)
        at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7280)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:552)
        at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:536)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamily
QueryImpl.java:201)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamily
QueryImpl.java:188)
        at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:55)
        ... 28 more

Cassandra's log:

 WARN [ScheduledTasks:1] 2013-02-07 17:46:24,935 StorageService.java (line 2855) Flushing
CFS(Keyspace='R
ecommender', ColumnFamily='edgestore') to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-02-07 17:46:24,935 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-edgestore@2032656921(3689547/40713915 serialized/live bytes, 165613 ops)
 INFO [FlushWriter:237] 2013-02-07 17:46:24,961 Memtable.java (line 264) Writing Memtable-edgestore@20326
56921(3689547/40713915 serialized/live bytes, 165613 ops)
 INFO [FlushWriter:237] 2013-02-07 17:46:29,798 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-7-Data.db (2143580 bytes) for
commitlog pos
ition ReplayPosition(segmentId=1360013782638, position=19121495)
 INFO [MemoryMeter:1] 2013-02-07 17:49:39,624 Memtable.java (line 213) CFS(Keyspace='Recommender',
Column
Family='edgestore') liveRatio is 9.312177865096587 (just-counted was 9.312177865096587). 
calculation too
k 868ms for 11757 columns
 INFO [Thrift:1977] 2013-02-07 17:58:01,292 MigrationManager.java (line 174) Drop Keyspace
'Recommender'
 INFO [MigrationStage:1] 2013-02-07 17:58:01,294 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-schema_keyspaces@937719474(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,362 Memtable.java (line 264) Writing Memtable-schema_keyspace
s@937719474(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,404 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-96-Data.db (61 bytes) for
commitlog pos
ition ReplayPosition(segmentId=1360013782638, position=21758272)
 INFO [MigrationStage:1] 2013-02-07 17:58:01,513 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-schema_columnfamilies@1656096012(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,513 Memtable.java (line 264) Writing Memtable-schema_columnfa
milies@1656096012(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,519 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-302-Data.db (61
bytes) for co
mmitlog position ReplayPosition(segmentId=1360013782638, position=21758674)
 INFO [CompactionExecutor:442] 2013-02-07 17:58:01,531 CompactionTask.java (line 109) Compacting
[SSTable
Reader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-96-Data.db'),
 SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-95-Da
ta.db'), SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-
hf-94-Data.db'), SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_ke
yspaces-hf-93-Data.db')]
 INFO [MigrationStage:1] 2013-02-07 17:58:01,596 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-schema_columns@1442995080(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,597 Memtable.java (line 264) Writing Memtable-schema_columns@
1442995080(0/0 serialized/live bytes, 1 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,722 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/system/schema_columns/system-schema_columns-hf-63-Data.db (61 bytes) for commitlog
positio
n ReplayPosition(segmentId=1360013782638, position=21759634)
 INFO [CompactionExecutor:442] 2013-02-07 17:58:01,746 CompactionTask.java (line 221) Compacted
to [/home
/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-97-Data.db,].  870
to 309 (~35%
 of original) bytes for 2 keys at 0.001371MB/s.  Time: 215ms.
 INFO [MigrationStage:1] 2013-02-07 17:58:01,862 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-titan_ids@1923920972(2304/2880 serialized/live bytes, 36 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,862 Memtable.java (line 264) Writing Memtable-titan_ids@19239
20972(2304/2880 serialized/live bytes, 36 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,914 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/Recommender/titan_ids/Recommender-titan_ids-hf-1-Data.db (870 bytes) for commitlog
positio
n ReplayPosition(segmentId=1360013782638, position=21759634)
 INFO [MigrationStage:1] 2013-02-07 17:58:01,916 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-propertyindex_lock_@540683493(840/1050 serialized/live bytes, 28 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:01,971 Memtable.java (line 264) Writing Memtable-propertyindex_l
ock_@540683493(840/1050 serialized/live bytes, 28 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:02,027 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/Recommender/propertyindex_lock_/Recommender-propertyindex_lock_-hf-1-Data.db
(650 bytes) f
or commitlog position ReplayPosition(segmentId=1360013782638, position=21759634)
 INFO [MigrationStage:1] 2013-02-07 17:58:02,047 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-edgestore@2005322361(1191757/13872316 serialized/live bytes, 53956 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:02,050 Memtable.java (line 264) Writing Memtable-edgestore@20053
22361(1191757/13872316 serialized/live bytes, 53956 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:02,134 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-8-Data.db (720306 bytes) for
commitlog posi
tion ReplayPosition(segmentId=1360013782638, position=21760036)
 INFO [CompactionExecutor:443] 2013-02-07 17:58:02,135 CompactionTask.java (line 109) Compacting
[SSTable
Reader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-6-Data.db'),
SSTa
bleReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-8-Data.db'),
S
STableReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-5-Data.db')
, SSTableReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-7-Data.d
b')]
 INFO [MigrationStage:1] 2013-02-07 17:58:02,160 ColumnFamilyStore.java (line 659) Enqueuing
flush of Mem
table-propertyindex@1052449648(243223/8141246 serialized/live bytes, 8387 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:02,160 Memtable.java (line 264) Writing Memtable-propertyindex@1
052449648(243223/8141246 serialized/live bytes, 8387 ops)
 INFO [FlushWriter:238] 2013-02-07 17:58:02,223 Memtable.java (line 305) Completed flushing
/home/apollo/
cassandra/data/Recommender/propertyindex/Recommender-propertyindex-hf-3-Data.db (301191 bytes)
for commit
log position ReplayPosition(segmentId=1360013782638, position=21760036)
 INFO [CompactionExecutor:443] 2013-02-07 17:58:05,032 CompactionTask.java (line 221) Compacted
to [/home
/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-9-Data.db,].  17,110,084
to 16,336,
307 (~95% of original) bytes for 78,555 keys at 5.377810MB/s.  Time: 2,897ms.
 INFO [CompactionExecutor:444] 2013-02-07 21:36:21,419 AutoSavingCache.java (line 268) Saved
KeyCache (35
939 items) in 31 ms
 INFO [CompactionExecutor:445] 2013-02-08 01:36:21,435 AutoSavingCache.java (line 268) Saved
KeyCache (35
939 items) in 37 ms
 INFO [CompactionExecutor:446] 2013-02-08 05:36:21,447 AutoSavingCache.java (line 268) Saved
KeyCache (40
902 items) in 38 ms
 INFO [CompactionExecutor:447] 2013-02-08 09:36:21,478 AutoSavingCache.java (line 268) Saved
KeyCache (40
902 items) in 59 ms






Mime
View raw message