cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Harvey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2309) Scrub resulting in "Keys must be written in ascending order" exception
Date Wed, 23 Mar 2011 19:43:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010317#comment-13010317
] 

Jason Harvey commented on CASSANDRA-2309:
-----------------------------------------

bq. That looks a lot like "sstable2json couldn't find the Cassandra schema"

I grabbed some debug output. Looks like it can find the schema. It is dying just after it
loads the index:

{code}
 INFO 12:39:45,401 Loading settings from file:/etc/cassandra/cassandra.yaml
DEBUG 12:39:45,623 Syncing log with a period of 10000
 INFO 12:39:45,623 DiskAccessMode is standard, indexAccessMode is mmap
DEBUG 12:39:45,709 setting auto_bootstrap to false
DEBUG 12:39:45,853 Initializing system.IndexInfo
DEBUG 12:39:45,880 Starting CFS IndexInfo
DEBUG 12:39:45,895 key cache capacity for IndexInfo is 1
DEBUG 12:39:45,896 row cache capacity for IndexInfo is 0
DEBUG 12:39:45,899 Initializing system.Schema
DEBUG 12:39:45,901 Starting CFS Schema
 INFO 12:39:45,902 reading saved cache /var/lib/cassandra/saved_caches/system-Schema-KeyCache
DEBUG 12:39:45,903 completed reading (0 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-Schema-KeyCache
 INFO 12:39:45,920 Opening /var/lib/cassandra/data/system/Schema-f-9
DEBUG 12:39:45,920 Load statistics for /var/lib/cassandra/data/system/Schema-f-9
 INFO 12:39:45,935 JNA not found. Native methods will be disabled.
DEBUG 12:39:45,961 INDEX LOAD TIME for /var/lib/cassandra/data/system/Schema-f-9: 41 ms.
DEBUG 12:39:45,962 key cache contains 1/1 keys
DEBUG 12:39:45,962 adding /var/lib/cassandra/data/system/Schema-f-9 to list of files tracked
for system.Schema
DEBUG 12:39:45,962 row cache capacity for Schema is 0
DEBUG 12:39:45,963 Initializing system.Migrations
DEBUG 12:39:45,965 Starting CFS Migrations
 INFO 12:39:45,967 Opening /var/lib/cassandra/data/system/Migrations-f-9
DEBUG 12:39:45,967 Load statistics for /var/lib/cassandra/data/system/Migrations-f-9
DEBUG 12:39:45,969 INDEX LOAD TIME for /var/lib/cassandra/data/system/Migrations-f-9: 2 ms.
DEBUG 12:39:45,969 key cache contains 0/0 keys
DEBUG 12:39:45,970 adding /var/lib/cassandra/data/system/Migrations-f-9 to list of files tracked
for system.Migrations
DEBUG 12:39:45,970 key cache capacity for Migrations is 1
DEBUG 12:39:45,970 row cache capacity for Migrations is 0
DEBUG 12:39:45,971 Initializing system.LocationInfo
DEBUG 12:39:45,973 Starting CFS LocationInfo
 INFO 12:39:45,974 reading saved cache /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
DEBUG 12:39:45,975 completed reading (1 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-LocationInfo-KeyCache
 INFO 12:39:45,986 Opening /var/lib/cassandra/data/system/LocationInfo-f-290
DEBUG 12:39:45,986 Load statistics for /var/lib/cassandra/data/system/LocationInfo-f-290
DEBUG 12:39:45,989 INDEX LOAD TIME for /var/lib/cassandra/data/system/LocationInfo-f-290:
3 ms.
DEBUG 12:39:45,989 key cache contains 1/1 keys
 INFO 12:39:45,989 Opening /var/lib/cassandra/data/system/LocationInfo-f-291
DEBUG 12:39:45,989 Load statistics for /var/lib/cassandra/data/system/LocationInfo-f-291
DEBUG 12:39:45,992 INDEX LOAD TIME for /var/lib/cassandra/data/system/LocationInfo-f-291:
3 ms.
DEBUG 12:39:45,992 key cache contains 1/2 keys
DEBUG 12:39:45,992 adding /var/lib/cassandra/data/system/LocationInfo-f-290 to list of files
tracked for system.LocationInfo
DEBUG 12:39:45,993 adding /var/lib/cassandra/data/system/LocationInfo-f-291 to list of files
tracked for system.LocationInfo
DEBUG 12:39:45,993 row cache capacity for LocationInfo is 0
DEBUG 12:39:45,994 Initializing system.HintsColumnFamily
DEBUG 12:39:45,996 Starting CFS HintsColumnFamily
 INFO 12:39:45,997 reading saved cache /var/lib/cassandra/saved_caches/system-HintsColumnFamily-KeyCache
DEBUG 12:39:45,997 completed reading (1 ms; 1 keys) saved cache /var/lib/cassandra/saved_caches/system-HintsColumnFamily-KeyCache
DEBUG 12:39:45,998 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,998 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
DEBUG 12:39:45,999 not including compacted sstable HintsColumnFamily-71
 INFO 12:39:46,000 Opening /var/lib/cassandra/data/system/HintsColumnFamily-f-72
DEBUG 12:39:46,000 Load statistics for /var/lib/cassandra/data/system/HintsColumnFamily-f-72
DEBUG 12:39:46,002 INDEX LOAD TIME for /var/lib/cassandra/data/system/HintsColumnFamily-f-72:
2 ms.
DEBUG 12:39:46,003 key cache contains 1/1 keys
 INFO 12:39:46,003 Opening /var/lib/cassandra/data/system/HintsColumnFamily-f-73
DEBUG 12:39:46,003 Load statistics for /var/lib/cassandra/data/system/HintsColumnFamily-f-73
DEBUG 12:39:46,006 INDEX LOAD TIME for /var/lib/cassandra/data/system/HintsColumnFamily-f-73:
3 ms.
DEBUG 12:39:46,006 key cache contains 2/2 keys
DEBUG 12:39:46,006 adding /var/lib/cassandra/data/system/HintsColumnFamily-f-72 to list of
files tracked for system.HintsColumnFamily
DEBUG 12:39:46,007 adding /var/lib/cassandra/data/system/HintsColumnFamily-f-73 to list of
files tracked for system.HintsColumnFamily
DEBUG 12:39:46,007 row cache capacity for HintsColumnFamily is 0
 INFO 12:39:46,048 Loading schema version 09021612-3bac-11e0-b472-fc83263a611f
DEBUG 12:39:46,059 collecting 0 of 2147483647: Avro/Schema:false:2042@1298067089267
DEBUG 12:39:46,060 collecting 1 of 2147483647: reddit:false:2502@1298067089267
 WARN 12:39:46,396 Schema definitions were defined both locally and in cassandra.yaml. Definitions
in cassandra.yaml were ignored.
 INFO 12:39:46,415 Opening /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223
DEBUG 12:39:46,415 Load statistics for /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223
DEBUG 12:39:48,175 INDEX LOAD TIME for /var/lib/cassandra/data/reddit/snapshots/1300857807427-prescrub/permacache-f-161223:
1760 ms.
{
Exception in thread "main" java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.create(ColumnFamily.java:67)
        at org.apache.cassandra.io.sstable.SSTableReader.createColumnFamily(SSTableReader.java:583)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:110)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:67)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179)
        at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144)
        at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136)
        at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:293)
        at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:324)
        at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:337)
        at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:395)
{code}

> Scrub resulting in "Keys must be written in ascending order" exception
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-2309
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2309
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.7.3, 0.7.4
>            Reporter: Jason Harvey
>
> Getting the following when I try to scrub. The SSTable causing it is over a gig. Trying
to repro on smaller tables so I have something to upload.
> {code}
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,264 CompactionManager.java (line 543)
Reading row at 552076476
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 552)
row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
> DEBUG [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 581)
Index doublecheck: row 6c6173745f636f6d6d656e74735f74335f386c387633 is 168 bytes
>  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 108) Last
written key : DecoratedKey(125686934811414729670440675125192621396, 627975726c2833626333626339353363353762313133373331336461303233396438303534312c66692e676f73757065726d6f64656c2e636f6d2f70726f66696c65732f2f6170706c65747265713d3132373333393332313937363529)
>  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 109) Current
key : DecoratedKey(11163129555189411157074346827960371449, 6c6173745f636f6d6d656e74735f74335f386c387633)
>  INFO [CompactionExecutor:1] 2011-03-10 12:07:13,265 SSTableWriter.java (line 110) Writing
into file /var/lib/cassandra/data/reddit/permacache-tmp-f-168492-Data.db
>  WARN [CompactionExecutor:1] 2011-03-10 12:07:13,265 CompactionManager.java (line 599)
Non-fatal error reading row (stacktrace follows)
> java.io.IOException: Keys must be written in ascending order.
>         at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
>         at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>         at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>         at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message