cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Victor Trac (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12009) Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed
Date Tue, 14 Jun 2016 23:02:30 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330825#comment-15330825
] 

Victor Trac commented on CASSANDRA-12009:
-----------------------------------------

I don't think our problem is due to limits:
{code}
# cat /proc/16854/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             32768                32768                processes
Max open files            100000               100000               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       122664               122664               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 122664
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 900000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 122664
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
{code}



> Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-12009
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12009
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: # java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
> # uname -a
> Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa datastax-ddc
> datastax-ddc-3.5.0-1.noarch
>            Reporter: Victor Trac
>
> Running nodetool scrub fails after a day or two. Originally, the instance had 8GB of
RAM with 4GB of Xmx/Xms. After seeing the OOM error several times, I upgraded the box to a
24GB RAM box, giving 20GB to Xms/Xmx to see if it would finish the scrub operation. Both cases
failed with the same error as below (tried on both the CMS and G1 GC collectors):
> {code}
> INFO  [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 - system_traces.events
                     0,0
> DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - Convicting /10.240.0.4
with status removed - alive false
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,151 OutputHandler.java:57 - Error reading
row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
>         ... 24 common frames omitted
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,152 OutputHandler.java:52 - Row starting
at position 31397042159 is unreadable; skipping to next
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,257 OutputHandler.java:57 - Error reading
row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
>         ... 24 common frames omitted
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,258 OutputHandler.java:52 - Row starting
at position 31397303324 is unreadable; skipping to next
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:57 - Error reading
row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.5.0.jar:3.5Gq
>         at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_91]
>         at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-3.5.0.jar:3.5.0]
>         ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_91]
>         ... 24 common frames omitted
> WARN  [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:52 - Row starting
at position 31397313498 is unreadable; skipping to next
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message