cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joe Fox (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4832) AssertionError: keys must not be empty
Date Mon, 22 Oct 2012 06:44:14 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481209#comment-13481209
] 

Joe Fox commented on CASSANDRA-4832:
------------------------------------

I've run into the same symptoms as Chris, but while attempting to migrate data between a 1.1.1
cluster and 1.1.6.

Whilst using sstableloader to import the data, nodes would hit certain column indexes and
stop processing further requests once the assertion was thrown - restarting the node would
almost immediately throw the assertion again, and the node would just fail to rejoin the ring.

tpstats would show active flushwriter tasks but no further node activity.

We had to work around the issue by:

1. Removing all indexes from our target cluster schema
2. Importing the data via sstableloader
3. Scanning through relevant column families and inserting data into any empty indexed columns
4. Re-applying the indexes to the target cluster schema

Only then was the migration successful.

I'll also note that attempting to apply an index to a column which has null data will also
throw the cluster out of sync as the nodes which throw the assertion fail to migrate their
schemas properly

 INFO [Creating index: Transactions.TransactionsCountryCode] 2012-10-21 07:45:25,978 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512
serialized/live bytes, 762 ops)
 INFO [Creating index: Transactions.TransactionsStatus] 2012-10-21 07:45:25,980 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-Transactions.TransactionsStatus@673679943(38862/125966
serialized/live bytes, 762 ops)
 INFO [FlushWriter:1] 2012-10-21 07:45:25,987 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsCountryCode@1802367190(38862/169512
serialized/live bytes, 762 ops)
 INFO [FlushWriter:2] 2012-10-21 07:45:26,004 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsStatus@673679943(38862/125966
serialized/live bytes, 762 ops)
ERROR [FlushWriter:1] 2012-10-21 07:45:26,004 AbstractCassandraDaemon.java (line 135) Exception
in thread Thread[FlushWriter:1,5,main]
java.lang.AssertionError: Keys must not be empty
	at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
	at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:176)
	at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:295)
	at org.apache.cassandra.db.Memtable.access$600(Memtable.java:48)
	at org.apache.cassandra.db.Memtable$5.runMayThrow(Memtable.java:316)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
 INFO [FlushWriter:2] 2012-10-21 07:45:26,049 Memtable.java (line 305) Completed flushing
/var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsStatus-hf-2-Data.db
(35259 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
 INFO [Creating index: Transactions.TransactionsLastUpdateDate] 2012-10-21 07:45:26,313 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277
serialized/live bytes, 762 ops)
 INFO [FlushWriter:2] 2012-10-21 07:45:26,314 Memtable.java (line 264) Writing Memtable-Transactions.TransactionsLastUpdateDate@1912098049(38862/240277
serialized/live bytes, 762 ops)
 INFO [FlushWriter:2] 2012-10-21 07:45:26,743 Memtable.java (line 305) Completed flushing
/var/lib/cassandra/data/*******/Transactions/*******-Transactions.TransactionsLastUpdateDate-hf-2-Data.db
(37024 bytes) for commitlog position ReplayPosition(segmentId=1350805525722, position=0)
 INFO [main] 2012-10-21 07:45:27,052 CommitLogReplayer.java (line 272) Finished reading /var/lib/cassandra/commitlog/CommitLog-1350768744805.log
 INFO [main] 2012-10-21 07:45:27,054 ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-Versions@1851630436(83/103 serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-10-21 07:45:27,054 Memtable.java (line 264) Writing Memtable-Versions@1851630436(83/103
serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-10-21 07:45:27,061 Memtable.java (line 305) Completed flushing
/var/lib/cassandra/data/system/Versions/system-Versions-hf-1-Data.db (247 bytes) for commitlog
position ReplayPosition(segmentId=1350805525722, position=0)
                
> AssertionError: keys must not be empty
> --------------------------------------
>
>                 Key: CASSANDRA-4832
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4832
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.6
>         Environment: Debian 6.0.5
>            Reporter: Tristan Seligmann
>            Assignee: Tristan Seligmann
>            Priority: Minor
>              Labels: indexing
>             Fix For: 1.1.7
>
>         Attachments: FlushWriterKeyAssertionBlock.txt
>
>
> I'm getting errors like this logged:
>  INFO 07:08:32,104 Compacting [SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-114-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-113-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-110-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-108-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-106-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hd-107-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-112-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-109-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Fusion/quoteinfo/Fusion-quoteinfo.quoteinfo_search_value_idx-hf-111-Data.db')]
> ERROR 07:08:32,108 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.AssertionError: Keys must not be empty
>         at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:133)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:154)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> I'm not really sure when this started happening; they tend to be logged during a repair
but I can't reproduce the error 100% reliably.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message