cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom van der Woerdt (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-12125) ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 CassandraDaemon.java:185 - Exception in thread Thread[MemtableFlushWriter:4,5,main] java.lang.RuntimeException: Last written key DecoratedKey(XXXX.XX, XXXXX) >= current key DecoratedKey
Date Tue, 25 Oct 2016 07:40:59 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604512#comment-15604512
] 

Tom van der Woerdt edited comment on CASSANDRA-12125 at 10/25/16 7:40 AM:
--------------------------------------------------------------------------

This issue still exists in 3.0.9.

{noformat}
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [10..11]:
11
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [12..13]:
38
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [14..15]:
23
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [16..17]:
17
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [18..19]:
12
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [20..21]:
9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [22..29]:
6
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 Validator.java:235 - Validated 1868
partitions for bb1f9dd0-98db-11e6-ad5a-89fa523ea316.  Partition sizes are:
INFO  [AntiEntropyStage:1] 2016-10-23 04:46:59,703 Validator.java:274 - [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316]
Sending completed merkle tree to /10.0.0.6 for myschema.mytable2
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 -    [17232..387341]:
99
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 -  [387342..1097582]:
16
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1097583..1807823]:
3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1807824..2518064]:
7
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [2518065..3959516]:
3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 CompactionManager.java:1135 - Validation
finished in 793 msec, for [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316 on myschema/mytable2,
[(-6682001123294953015,-6681421039287311765]]]
DEBUG [CompactionExecutor:51358] 2016-10-23 04:47:00,499 CompactionTask.java:218 - Compacted
(bb4d3d80-98db-11e6-b4a3-2b509f67851f) 2 sstables to [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big,]
to level=0.  31,664,366 bytes to 31,653,685 (~99% of original) in 1,592ms = 18.961876MB/s.
 0 total partitions merged to 1,888.  Partition merge counts were {1:2063, }
DEBUG [GossipStage:1] 2016-10-23 04:47:00,705 FailureDetector.java:456 - Ignoring interval
time of 2065449328 for /10.0.0.1
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval
time of 2065577694 for /10.0.0.2
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval
time of 2065549822 for /10.0.0.3
DEBUG [Thread-6] 2016-10-23 04:47:01,201 StorageProxy.java:2246 - Schemas are in agreement.
DEBUG [GossipStage:1] 2016-10-23 04:47:01,635 FailureDetector.java:456 - Ignoring interval
time of 2543137972 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval
time of 2218246254 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval
time of 2218388771 for /10.0.0.5
DEBUG [GossipStage:1] 2016-10-23 04:47:04,549 FailureDetector.java:456 - Ignoring interval
time of 2000269752 for /10.0.0.2
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,281 RepairMessageVerbHandler.java:116 - Validating
ValidationRequest{gcBefore=1474606025} org.apache.cassandra.repair.messages.ValidationRequest@57e4b3c9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,281 StorageService.java:2955 - Forcing
flush on keyspace myschema, CF mytable1
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,282 ColumnFamilyStore.java:850 - Enqueuing
flush of mytable1: 28142 (0%) on-heap, 16292220 (0%) off-heap
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,282 Memtable.java:368 - Writing Memtable-mytable1@1772797855(3.616MiB
serialized bytes, 47 ops, 0%/0% of on/off-heap limit)
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,285 RepairMessageVerbHandler.java:133 - Syncing
SyncRequest{initiator=/10.0.0.6, src=/10.0.0.7, dst=/10.0.0.1, ranges=[(-6681996591388643318,-6681992059482333621],
(-6681946740419236648,-6681942208512926951], (-6681933144700307557,-6681928612793997859],
(-6681915017075068768,-6681910485168759070], (-6681896889449829979,-6681892357543520281],
(-6681887825637210584,-6681883293730900887], (-6681842506574113612,-6681837974667803914],
(-6681837974667803914,-6681833442761494217], (-6681815315136255428,-6681810783229945731],
(-6681810783229945731,-6681801719417326336], (-6681783591792087547,-6681779059885777850],
(-6681769996073158456,-6681765464166848758], (-6681747336541609969,-6681742804635300272],
(-6681720145103751785,-6681715613197442088], (-6681634038883867537,-6681629506977557840],
(-6681629506977557840,-6681620443164938445], (-6681593251727080262,-6681584187914460867],
(-6681570592195531776,-6681566060289222078], (-6681552464570292987,-6681547932663983289],
(-6681547932663983289,-6681543400757673592], (-6681520741226125106,-6681516209319815409],
(-6681493549788266922,-6681484485975647528], (-6681448230725169950,-6681443698818860253]]}
org.apache.cassandra.repair.messages.SyncRequest@5fbf8b2c
INFO  [AntiEntropyStage:1] 2016-10-23 04:47:05,286 StreamingRepairTask.java:58 - [streaming
task #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Performing streaming repair of 23 ranges with
/10.0.0.1
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,286 ColumnFamilyStore.java:850 - Enqueuing
flush of mytable2: 1695 (0%) on-heap, 19377 (0%) off-heap
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,286 Memtable.java:368 - Writing Memtable-mytable2@1296803980(19.105KiB
serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,322 Memtable.java:401 - Completed flushing
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db
(19.019KiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985)
DEBUG [CompactionExecutor:51352] 2016-10-23 04:47:05,366 CompactionTask.java:146 - Compacting
(bf274360-98db-11e6-b4a3-2b509f67851f) [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db:level=0,
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big-Data.db:level=0,
]
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,533 Memtable.java:401 - Completed flushing
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/mc-422108-big-Data.db
(3.615MiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985)
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,695 Memtable.java:368 - Writing Memtable-mytable1.mytable1_my_idx@185942079(1.187KiB
serialized bytes, 28 ops, 0%/0% of on/off-heap limit)
ERROR [MemtableFlushWriter:5503] 2016-10-23 04:47:05,757 CassandraDaemon.java:205 - Exception
in thread Thread[MemtableFlushWriter:5503,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(1byAcm-002Hsb-EL, 31627941636d2d3030324873622d454c)
>= current key DecoratedKey(^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@, 31627941656e2d3030304171552d554b)
writing into /var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/.mytable1_my_idx/mc-16240-big-Data.db
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.Memtable.flush(Memtable.java:332) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
{noformat}

After this, the commitlog directory grew a lot (files in the directory weren't cleaned up
anymore, because MemtableFlushWriter died), and the node ran out of space :)

Again on a table with a secondary index, so that's likely related.

Schema :
{noformat}
CREATE TABLE myschema.mytable1 (
    id bigint PRIMARY KEY,
    field1 blob,
    field2 ascii
) WITH bloom_filter_fp_chance = 0.1
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_in_kb': '1024', 'class': 'org.apache.cassandra.io.compress.DeflateCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 2592000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX mytable1_my_idx ON myschema.mytable1 (field2);
{noformat}


was (Author: tvdw):
This issue still exists in 3.0.9.

{noformat}
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [10..11]:
11
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [12..13]:
38
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [14..15]:
23
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [16..17]:
17
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [18..19]:
12
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [20..21]:
9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [22..29]:
6
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 Validator.java:235 - Validated 1868
partitions for bb1f9dd0-98db-11e6-ad5a-89fa523ea316.  Partition sizes are:
INFO  [AntiEntropyStage:1] 2016-10-23 04:46:59,703 Validator.java:274 - [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316]
Sending completed merkle tree to /10.0.0.6 for myschema.mytable2
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 -    [17232..387341]:
99
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 -  [387342..1097582]:
16
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1097583..1807823]:
3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [1807824..2518064]:
7
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 EstimatedHistogram.java:304 - [2518065..3959516]:
3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 CompactionManager.java:1135 - Validation
finished in 793 msec, for [repair #bb1f9dd0-98db-11e6-ad5a-89fa523ea316 on myschema/mytable2,
[(-6682001123294953015,-6681421039287311765]]]
DEBUG [CompactionExecutor:51358] 2016-10-23 04:47:00,499 CompactionTask.java:218 - Compacted
(bb4d3d80-98db-11e6-b4a3-2b509f67851f) 2 sstables to [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big,]
to level=0.  31,664,366 bytes to 31,653,685 (~99% of original) in 1,592ms = 18.961876MB/s.
 0 total partitions merged to 1,888.  Partition merge counts were {1:2063, }
DEBUG [GossipStage:1] 2016-10-23 04:47:00,705 FailureDetector.java:456 - Ignoring interval
time of 2065449328 for /10.0.0.1
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval
time of 2065577694 for /10.0.0.2
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 - Ignoring interval
time of 2065549822 for /10.0.0.3
DEBUG [Thread-6] 2016-10-23 04:47:01,201 StorageProxy.java:2246 - Schemas are in agreement.
DEBUG [GossipStage:1] 2016-10-23 04:47:01,635 FailureDetector.java:456 - Ignoring interval
time of 2543137972 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval
time of 2218246254 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 - Ignoring interval
time of 2218388771 for /10.0.0.5
DEBUG [GossipStage:1] 2016-10-23 04:47:04,549 FailureDetector.java:456 - Ignoring interval
time of 2000269752 for /10.0.0.2
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,281 RepairMessageVerbHandler.java:116 - Validating
ValidationRequest{gcBefore=1474606025} org.apache.cassandra.repair.messages.ValidationRequest@57e4b3c9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,281 StorageService.java:2955 - Forcing
flush on keyspace myschema, CF mytable1
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,282 ColumnFamilyStore.java:850 - Enqueuing
flush of mytable1: 28142 (0%) on-heap, 16292220 (0%) off-heap
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,282 Memtable.java:368 - Writing Memtable-mytable1@1772797855(3.616MiB
serialized bytes, 47 ops, 0%/0% of on/off-heap limit)
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,285 RepairMessageVerbHandler.java:133 - Syncing
SyncRequest{initiator=/10.0.0.6, src=/10.0.0.7, dst=/10.0.0.1, ranges=[(-6681996591388643318,-6681992059482333621],
(-6681946740419236648,-6681942208512926951], (-6681933144700307557,-6681928612793997859],
(-6681915017075068768,-6681910485168759070], (-6681896889449829979,-6681892357543520281],
(-6681887825637210584,-6681883293730900887], (-6681842506574113612,-6681837974667803914],
(-6681837974667803914,-6681833442761494217], (-6681815315136255428,-6681810783229945731],
(-6681810783229945731,-6681801719417326336], (-6681783591792087547,-6681779059885777850],
(-6681769996073158456,-6681765464166848758], (-6681747336541609969,-6681742804635300272],
(-6681720145103751785,-6681715613197442088], (-6681634038883867537,-6681629506977557840],
(-6681629506977557840,-6681620443164938445], (-6681593251727080262,-6681584187914460867],
(-6681570592195531776,-6681566060289222078], (-6681552464570292987,-6681547932663983289],
(-6681547932663983289,-6681543400757673592], (-6681520741226125106,-6681516209319815409],
(-6681493549788266922,-6681484485975647528], (-6681448230725169950,-6681443698818860253]]}
org.apache.cassandra.repair.messages.SyncRequest@5fbf8b2c
INFO  [AntiEntropyStage:1] 2016-10-23 04:47:05,286 StreamingRepairTask.java:58 - [streaming
task #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Performing streaming repair of 23 ranges with
/10.0.0.1
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,286 ColumnFamilyStore.java:850 - Enqueuing
flush of mytable2: 1695 (0%) on-heap, 19377 (0%) off-heap
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,286 Memtable.java:368 - Writing Memtable-mytable2@1296803980(19.105KiB
serialized bytes, 3 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,322 Memtable.java:401 - Completed flushing
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db
(19.019KiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985)
DEBUG [CompactionExecutor:51352] 2016-10-23 04:47:05,366 CompactionTask.java:146 - Compacting
(bf274360-98db-11e6-b4a3-2b509f67851f) [/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db:level=0,
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big-Data.db:level=0,
]
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,533 Memtable.java:401 - Completed flushing
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/mc-422108-big-Data.db
(3.615MiB) for commitlog position ReplayPosition(segmentId=1474469138946, position=165305985)
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,695 Memtable.java:368 - Writing Memtable-mytable1.mytable1_my_idx@185942079(1.187KiB
serialized bytes, 28 ops, 0%/0% of on/off-heap limit)
ERROR [MemtableFlushWriter:5503] 2016-10-23 04:47:05,757 CassandraDaemon.java:205 - Exception
in thread Thread[MemtableFlushWriter:5503,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(1byAcm-002Hsb-EL, 31627941636d2d3030324873622d454c)
>= current key DecoratedKey(^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@, 31627941656e2d3030304171552d554b)
writing into /var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/.mytable1_my_idx/mc-16240-big-Data.db
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45)
~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.Memtable.flush(Memtable.java:332) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-3.0.9.jar:3.0.9]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
{noformat}

After this, the commitlog directory grew a lot (files in the directory weren't cleaned up
anymore, because MemtableFlushWriter died), and the node ran out of space :)

Again on a table with a secondary index, so that's likely related.

Schema :
{noformat}
CREATE TABLE bizmail.pgmailstorage (
    id bigint PRIMARY KEY,
    field1 blob,
    field2 ascii
) WITH bloom_filter_fp_chance = 0.1
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_in_kb': '1024', 'class': 'org.apache.cassandra.io.compress.DeflateCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 2592000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';
CREATE INDEX mytable1_my_idx ON myschema.mytable1 (field2);
{noformat}

> ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 CassandraDaemon.java:185 - Exception
in thread Thread[MemtableFlushWriter:4,5,main]  java.lang.RuntimeException: Last written key
DecoratedKey(XXXX.XX, XXXXX) >= current key DecoratedKey
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12125
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12125
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: RHEL-6.5 64-bit Apache Cassandra 2.2.5v
>            Reporter: Relish Chackochan
>             Fix For: 2.2.x
>
>
> We are running on RHEL-6.5 64-bit with Apache Cassandra 2.2.5v on 4 node cluster and
getting the following error on multiple node while running the repair job and when getting
the error repair job is hang.
> Can some one help to identify the issue.
> {code}
> ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 CassandraDaemon.java:185 - Exception
in thread Thread[MemtableFlushWriter:4,5,main]
> java.lang.RuntimeException: Last written key DecoratedKey(1467371986.8870, 313436373337313938362e38383730)
>= current key DecoratedKey(, 313436373337323030312e38383730) writing into /opt/cassandra/data/proddb/log_data1-0a5092a0a4fa11e5872fc1ce0a46dc27/.maxdatetimeindex_idx/tmp-la-470-big-Data.db
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message