cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ricardo Bartolome (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13663) Cassandra 3.10 crashes without dump
Date Fri, 03 Nov 2017 19:15:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238205#comment-16238205
] 

Ricardo Bartolome commented on CASSANDRA-13663:
-----------------------------------------------

We can't easily share the JVM crash log because it contains much information about keyspaces
that expose customer names. So far we can share this:
{code}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8e9fcf75ac, pid=16151, tid=0x00007f8b7aa93700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed
oops)
# Problematic frame:
# J 21889 C2 org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;)Lorg/apache/cassandra/db/RowIndexEntry;
(361 bytes) @ 0x00007f8e9fcf75ac [0x00007f8e9fcf42c0+0x32ec]
#
# Core dump written. Default location: //core or core.16151
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x000000000cd77920):  JavaThread "PerDiskMemtableFlushWriter_0:140" daemon
[_thread_in_Java, id=28952, stack(0x00007f8b7aa53000,0x00007f8b7aa94000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000002fbb07de
{code}

> Cassandra 3.10 crashes without dump
> -----------------------------------
>
>                 Key: CASSANDRA-13663
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13663
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Matthias Otto
>            Priority: Minor
>         Attachments: 2017-07-04 10_48_34-CloudWatch Management Console.png, RamUsageExamle1.png,
RamUsageExample2.png, cassandra debug.log, cassandra system.log
>
>
> Hello. My company runs a 5 node Cassandra cluster. For the last few weeks, we have had
a sporadic issue where one of the servers crashes without creating a dump file and without
any error messages in the logs. If one restarts the service (which we have by now scripted
to happen automatically), the servers resumes work with no complaint.
> Log files of the time of the last crash are attached, thou again they do not log any
crash happening.
> Regarding out setup, we are running these servers on AMazon AWS, with 3 volumes per server,
one for the system, one for data and one for the commitlog. When a crash happens, we can observe
a sudden spike of read activity on the commitlog volume. All of these have ample free space.
Aspecially the system volume has more then enough free space so that a dump could be written.
> The servers are Ubuntu 16.04 servers and Cassandra is installed from the apt-get packet
for version 3.10.
> It is worth noting that these crashes happen more often when nodetool is running either
repair job or a backup job, but this is by no means always the case. As for frequency, we
have had about 1-2 crashes per week for the last month.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message