cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Kania (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-11201) Compaction memory fault in 3.0.3
Date Sat, 20 Feb 2016 23:59:18 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Kania updated CASSANDRA-11201:
------------------------------------
    Description: 
I have been encountering the following errors periodically on the system:

ERROR [CompactionExecutor:6] 2016-02-20 16:54:09,069 CassandraDaemon.java:195 - Exception
in thread Thread[CompactionExecutor:6,1,main]
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled
Java code
        at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:376)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:175)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:280)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_65]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

This problem persisted after several reboots and even when most other applications on the
system were terminated to provide more memory availability.

The problem also occurs when running 'nodetool compact'.

  was:
I have been encountering the following errors periodically on the system:

ERROR [CompactionExecutor:6] 2016-02-20 16:54:09,069 CassandraDaemon.java:195 - Exception
in thread Thread[CompactionExecutor:6,1,main]
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled
Java code
        at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:376)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:175)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:280)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_65]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

This problem persisted after several reboots and even when most other applications on the
system were terminated to provide more memory availability.


> Compaction memory fault in 3.0.3
> --------------------------------
>
>                 Key: CASSANDRA-11201
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11201
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: debian jesse latest release, updated Feb. 20th
>            Reporter: Jason Kania
>
> I have been encountering the following errors periodically on the system:
> ERROR [CompactionExecutor:6] 2016-02-20 16:54:09,069 CassandraDaemon.java:195 - Exception
in thread Thread[CompactionExecutor:6,1,main]
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation
in compiled Java code
>         at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:376)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:175)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:280)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
~[apache-cassandra-3.0.3.jar:3.0.3]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_65]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_65]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> This problem persisted after several reboots and even when most other applications on
the system were terminated to provide more memory availability.
> The problem also occurs when running 'nodetool compact'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message