cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cameron Zemek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
Date Fri, 09 Sep 2016 06:02:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15476024#comment-15476024
] 

Cameron Zemek commented on CASSANDRA-12590:
-------------------------------------------

With added extra logging

{code:title=BigTableWriter.java|borderStyle=solid}
    protected long beforeAppend(DecoratedKey decoratedKey)
    {
        assert decoratedKey != null : "Keys must not be null"; // empty keys ARE allowed b/c
of indexed column values
        if (lastWrittenKey != null && lastWrittenKey.compareTo(decoratedKey) >=
0) {
            ByteBuffer key = decoratedKey.getKey();
            if (decoratedKey.getToken() instanceof LocalPartitioner.LocalToken) {
                LocalPartitioner.LocalToken token = (LocalPartitioner.LocalToken) decoratedKey.getToken();
                logger.warn("BigTableWriter::beforeAppend (" + (token == null ? "null" : ByteBufferUtil.bytesToHex(token.getTokenValue()))
+ ", " + (key == null ? "null" : ByteBufferUtil.bytesToHex(key)) + ")");
            } else {
                logger.warn("BigTableWriter::beforeAppend " + (key == null ? "null" : ByteBufferUtil.bytesToHex(key)));
            }
            throw new RuntimeException("Writing into " + getFilename() + " Last written key
" + lastWrittenKey + " >= current key " + decoratedKey);
        }
        return (lastWrittenKey == null) ? 0 : dataFile.position();
    }
{code}

got the following

{noformat}
Sep 06 16:58:54 ip-10-222-128-182.eu-west-1.compute.internal cassandra[7338]: WARN  o.a.c.i.s.format.big.BigTableWriter
BigTableWriter::beforeAppend (00000000000000000000007657683449634a374136764138703051562d6a48697165793065506354,
7a433144785f615a6e36417657683449634a374136764138703051562d6a48697165793065506354)
Sep 06 18:26:05 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]: WARN  o.a.c.i.s.format.big.BigTableWriter
BigTableWriter::beforeAppend (000000608fa527000000003578666a493271484e48505f514c574b544b486a454c555a6661316164,
4335485f61584e2d4c62593578666a493271484e48505f514c574b544b486a454c555a6661316164)
Sep 08 13:24:01 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]: WARN  o.a.c.i.s.format.big.BigTableWriter
BigTableWriter::beforeAppend (0000000000000000000000494b324c5268554f4c546852554143795f31634d724e385f4975797873,
634c5155767a77494a5f32494b324c5268554f4c546852554143795f31634d724e385f4975797873)
{noformat}

So first 11 bytes of the LocalToken ByteBuffer have been corrupted.

> Segfault reading secondary index
> --------------------------------
>
>                 Key: CASSANDRA-12590
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local Write-Read Paths
>         Environment: Occurs on Cassandra 3.5 and 3.7
>            Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2 org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
(53 bytes) @ 0x00007fd7354749b7 [0x00007fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I (9 bytes)
@ 0x00007fd7351b35b8 [0x00007fd7351b3440+0x178]
> J 14205 C2 java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
(142 bytes) @ 0x00007fd736404dd8 [0x00007fd736404cc0+0x118]
> J 17764 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
(635 bytes) @ 0x00007fd736e09638 [0x00007fd736e08720+0xf18]
> J 17808 C2 org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
(68 bytes) @ 0x00007fd736e01a48 [0x00007fd736e012a0+0x7a8]
> J 14217 C2 org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
(219 bytes) @ 0x00007fd736417c1c [0x00007fd736416fa0+0xc7c]
> J 14585 C2 org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
(337 bytes) @ 0x00007fd736541e6c [0x00007fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V (48 bytes)
@ 0x00007fd7357957b4 [0x00007fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ 0x00007fd735938d8c
[0x00007fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message