cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacek Lewandowski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8332) Null pointer after droping keyspace
Date Fri, 21 Nov 2014 12:52:35 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220863#comment-14220863
] 

Jacek Lewandowski commented on CASSANDRA-8332:
----------------------------------------------

I can see the problem with dropping keyspace right after write-intensive operations in 2.0.11.

The way to reproduce: 
1. clean data directory
2. start C*
3. run subsequent trials 

It happens after the second trial or later - depending on the amount of data being written
during a single trial. However, I couldn't reproduce it after a single trial. 

Now, the trial is:
1. create keyspace and table (bigint pk, 5 x text)
2. write *b* batches of *n* inserts in parallel (using Java Driver 2.1.2, protocol V2) with
*p* workers
3. drop keyspace
4. assert that the keyspace was really dropped by getting metadata from Cluster object

The procedure break in step 1, because asserting Cluster metadata is useless - it says the
keyspace is missing. However, in the next trial, when it tries to create the keyspace again,
it says that the keyspace already exists. Notice, that "drop keyspace" statement doesn't throw
any exception, either in Java Driver or in CQLSH.  

I tried different values for b, n and p, for example b=50k, n=50, p=6 and each text field
is 30 bytes long (10 x 24-bit utf characters)

System log shows various errors. Some of them occur even during step 2 in the second trial:
{noformat}
ERROR 13:01:00,422 Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/ks/tab/ks-tab-jb-2-Data.db
(No such file or directory)
at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1366)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1172)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1184)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:272)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:278)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/ks/tab/ks-tab-jb-2-Data.db
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:34)
at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:48)
... 17 more
{noformat}

Very simiar errors are added to the system log when the keyspace is tried to be dropped. However,
in this case I also noticed the following exceptions:
{noformat}
INFO 12:43:35,006 Compaction interrupted: Compaction@3e922a80-7599-3e0e-b804-6b32581a18d9(ks,
tab, 1579024/110473935)bytes
ERROR 12:43:35,007 Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExectionException: java.lang.RuntimeException:
Tried to hard link to file that does not exist /var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:285)
at org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:259)
at org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:79)
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
at com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448)
at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190)
at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223)
at com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103)
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:321)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Caused by:java.util.concurrent.ExecutionException: java.lang.RuntimeException: Tried to hard
link to file that does not exist /var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 19 more

Caused by: java.lang.RuntimeException: Tried to hard link to file that does not exist /var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:77)
at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1226)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1822)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1855)
at org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:404)
at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:288)
at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:184)
at org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:303)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{noformat}

and

{noformat}
java.lang.AssertionError
at org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
at org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:631)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1366)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1172)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1184)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:272)
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:278)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}

Notice, that there are no "null" in the exception. Also - this exception was thrown a little
bit later (few seconds) after "drop keyspace" command was tried. 


> Null pointer after droping keyspace
> -----------------------------------
>
>                 Key: CASSANDRA-8332
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8332
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Chris Lohfink
>            Assignee: T Jake Luciani
>            Priority: Minor
>             Fix For: 2.1.3
>
>
> After dropping keyspace, sometimes I see this in logs:
> {code}
> ERROR 03:40:29 Exception in thread Thread[CompactionExecutor:2,1,main]
> java.lang.AssertionError: null
> 	at org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:1142)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1896)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:68)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1681)
~[main/:na]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1693)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:181)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(WrappingCompactionStrategy.java:320)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:340)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:151)
~[main/:na]
> 	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[main/:na]
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:233)
~[main/:na]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_71]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_71]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> {code}
> Minor issue since doesn't really affect anything, but the error makes it look like somethings
wrong.  Seen on 2.1 branch (1b21aef8152d96a180e75f2fcc5afad9ded6c595), not sure how far back
(may be post 2.1.2).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message