cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua McKenzie (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-8337) mmap underflow during validation compaction
Date Wed, 26 Nov 2014 19:07:13 GMT


Joshua McKenzie updated CASSANDRA-8337:
    Attachment: 8337_v1.txt

Attaching a patch that will print out the path of the corrupt sstable on either a stop_paranoid
policy failure or die policy in JVMStabilityInspector.  If you can run with the above patch
on a test cluster it should tell us which files you're having trouble with.

If we could get one of those attached to this ticket that would be a big help, given that
scrub reports that the tables are ok.  If not, that's completely understandable, but thus
far I've had no luck reproducing corrupted data like this.

> mmap underflow during validation compaction
> -------------------------------------------
>                 Key: CASSANDRA-8337
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alexander Sterligov
>            Assignee: Joshua McKenzie
>         Attachments: 8337_v1.txt, thread_dump
> During full parallel repair I often get errors like the following
> {quote}
> [2014-11-19 01:02:39,355] Repair session 116beaf0-6f66-11e4-afbb-c1c082008cbe for range
(3074457345618263602,-9223372036854775808] failed with error org.apache.cassandra.exceptions.RepairException:
[repair #116beaf0-6f66-11e4-afbb-c1c082008cbe on iss/target_state_history, (3074457345618263602,-9223372036854775808]]
Validation failed in /
> {quote}
> At the log of the node there are always same exceptions:
> {quote}
> ERROR [ValidationExecutor:2] 2014-11-19 01:02:10,847 -
JVM state determined to be unstable.  Exiting forcefully due to:
> mmap segment
underflow; remaining is 15 but 47 requested
>         at
>         at
>         at
>         at
>         at
>         at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(
>         at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(
>         at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(
>         at org.apache.cassandra.db.compaction.CompactionManager.access$600(
>         at org.apache.cassandra.db.compaction.CompactionManager$
>         at ~[na:1.7.0_51]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at [na:1.7.0_51]
> Caused by: mmap segment underflow; remaining is 15 but 47 requested
>         at
>         at ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(
>         at
>         ... 13 common frames omitted
> {quote}
> Now i'm using die disk_failure_policy to determine such conditions faster, but I get
them even with stop policy.
> Streams related to host with such exception are hanged. Thread dump is attached. Only
restart helps.
> After retry I get errors from other nodes.
> scrub doesn't help and report that sstables are ok.
> Sequential repairs doesn't cause such exceptions.
> Load is about 1000 write rps and 50 read rps per node.

This message was sent by Atlassian JIRA

View raw message