cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Haddad <...@jonhaddad.com>
Subject Re: Datafile Corruption
Date Thu, 08 Aug 2019 16:34:46 GMT
Any chance you're using NVMe with an older Linux kernel?  I've seen a *lot*
filesystem errors from using older CentOS versions.  You'll want to be
using a version > 4.15.

On Thu, Aug 8, 2019 at 9:31 AM Philip Ó Condúin <philipoconduin@gmail.com>
wrote:

> *@Jeff *- If it was hardware that would explain it all, but do you think
> it's possible to have every server in the cluster with a hardware issue?
> The data is sensitive and the customer would lose their mind if I sent it
> off-site which is a pity cause I could really do with the help.
> The corruption is occurring irregularly on every server and instance and
> column family in the cluster.  Out of 72 instances, we are getting maybe 10
> corrupt files per day.
> We are using vnodes (256) and it is happening in both DC's
>
> *@Asad *- internode compression is set to ALL on every server.  I have
> checked the packets for the private interconnect and I can't see any
> dropped packets, there are dropped packets for other interfaces, but not
> for the private ones, I will get the network team to double-check this.
> The corruption is only on the application schema, we are not getting
> corruption on any system or cass keyspaces.  Corruption is happening in
> both DC's.  We are getting corruption for the 1 application schema we have
> across all tables in the keyspace, it's not limited to one table.
> Im not sure why the app team decided to not use default compression, I
> must ask them.
>
>
>
> I have been checking the /var/log/messages today going back a few weeks
> and can see a serious amount of broken pipe errors across all servers and
> instances.
> Here is a snippet from one server but most pipe errors are similar:
>
> Jul  9 03:00:08  cassandra: INFO  02:00:08 Writing
> Memtable-sstable_activity@1126262628(43.631KiB serialized bytes, 18072
> ops, 0%/0% of on/off-heap limit)
> Jul  9 03:00:13  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
> Jul  9 03:00:19  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
> Jul  9 03:00:22  cassandra: ERROR 02:00:22 Got an IOException during write!
> Jul  9 03:00:22  cassandra: java.io.IOException: Broken pipe
> Jul  9 03:00:22  cassandra: at sun.nio.ch.FileDispatcherImpl.write0(Native
> Method) ~[na:1.8.0_172]
> Jul  9 03:00:22  cassandra: at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_172]
> Jul  9 03:00:22  cassandra: at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_172]
> Jul  9 03:00:22  cassandra: at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> ~[na:1.8.0_172]
> Jul  9 03:00:22  cassandra: at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> ~[na:1.8.0_172]
> Jul  9 03:00:22  cassandra: at
> org.apache.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:165)
> ~[libthrift-0.9.2.jar:0.9.2]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.util.mem.Buffer.writeTo(Buffer.java:104)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.streamTo(FastMemoryOutputTransport.java:112)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.Message.write(Message.java:222)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.handleWrite(TDisruptorServer.java:598)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.processKey(TDisruptorServer.java:569)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.select(TDisruptorServer.java:423)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:22  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.run(TDisruptorServer.java:383)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:25  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
> Jul  9 03:00:30  cassandra: ERROR 02:00:30 Got an IOException during write!
> Jul  9 03:00:30  cassandra: java.io.IOException: Broken pipe
> Jul  9 03:00:30  cassandra: at sun.nio.ch.FileDispatcherImpl.write0(Native
> Method) ~[na:1.8.0_172]
> Jul  9 03:00:30  cassandra: at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_172]
> Jul  9 03:00:30  cassandra: at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_172]
> Jul  9 03:00:30  cassandra: at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> ~[na:1.8.0_172]
> Jul  9 03:00:30  cassandra: at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> ~[na:1.8.0_172]
> Jul  9 03:00:30  cassandra: at
> org.apache.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:165)
> ~[libthrift-0.9.2.jar:0.9.2]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.util.mem.Buffer.writeTo(Buffer.java:104)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.streamTo(FastMemoryOutputTransport.java:112)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.Message.write(Message.java:222)
> ~[thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.handleWrite(TDisruptorServer.java:598)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.processKey(TDisruptorServer.java:569)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.select(TDisruptorServer.java:423)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:30  cassandra: at
> com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.run(TDisruptorServer.java:383)
> [thrift-server-0.3.7.jar:na]
> Jul  9 03:00:31  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
> Jul  9 03:00:37  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
> Jul  9 03:00:43  kernel: fnic_handle_fip_timer: 8 callbacks suppressed
>
>
>
> On Thu, 8 Aug 2019 at 15:42, ZAIDI, ASAD A <az192g@att.com> wrote:
>
>> Did you check if packets are NOT being dropped for network interfaces
>> Cassandra instances are consuming (ifconfig –a) internode compression is
>> set for all endpoint – may be network is playing any role here?
>>
>> is this corruption limited so certain keyspace/table | DCs or is that
>> wide spread – the log snippet you shared it looked like only specific
>> keyspace/table is affected – is that correct?
>>
>> When you remove corrupted sstable of a certain table, I guess you
>> verifies all nodes for corrupted sstables for same table (may be with with
>> nodetool scrub tool) so to limit spread of corruptions – right?
>>
>> Just curious to know – you’re not using lz4/default compressor for all
>> tables there must be some reason for it.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Philip Ó Condúin [mailto:philipoconduin@gmail.com]
>> *Sent:* Thursday, August 08, 2019 6:20 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Datafile Corruption
>>
>>
>>
>> Hi All,
>>
>> Thank you so much for the replies.
>>
>> Currently, I have the following list that can potentially cause some sort
>> of corruption in a Cassandra cluster.
>>
>>    - Sudden Power cut  -  *We have had no power cuts in the datacenters*
>>    - Network Issues - *no network issues from what I can tell*
>>    - Disk full - *I don't think this is an issue for us, see disks
>>    below.*
>>    - An issue in Casandra version like Cassandra-13752 -* couldn't find
>>    any Jira issues similar to ours.*
>>    - Bit Flips -* we have compression enabled so I don't think this
>>    should be an issue.*
>>    - Repair during upgrade has caused corruption too -* we have not
>>    upgraded*
>>    - Dropping and adding columns with the same name but a different type
>>    - *I will need to ask the apps team how they are using the database.*
>>
>>
>>
>> Ok, let me try and explain the issue we are having, I am under a lot of
>> pressure from above to get this fixed and I can't figure it out.
>>
>> This is a PRE-PROD environment.
>>
>>    - 2 datacenters.
>>    - 9 physical servers in each datacenter
>>    - 4 Cassandra instances on each server
>>    - 72 Cassandra instances across the 2 data centres, 36 in site A, 36
>>    in site B.
>>
>>
>> We also have 2 Reaper Nodes we use for repair.  One reaper node in each
>> datacenter each running with its own Cassandra back end in a cluster
>> together.
>>
>> OS Details [Red Hat Linux]
>> cass_a@x 0 10:53:01 ~ $ uname -a
>> Linux x 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> cass_a@x 0 10:57:31 ~ $ cat /etc/*release
>> NAME="Red Hat Enterprise Linux Server"
>> VERSION="7.6 (Maipo)"
>> ID="rhel"
>>
>> Storage Layout
>> cass_a@xx 0 10:46:28 ~ $ df -h
>> Filesystem                         Size  Used Avail Use% Mounted on
>> /dev/mapper/vg01-lv_root            20G  2.2G   18G  11% /
>> devtmpfs                            63G     0   63G   0% /dev
>> tmpfs                               63G     0   63G   0% /dev/shm
>> tmpfs                               63G  4.1G   59G   7% /run
>> tmpfs                               63G     0   63G   0% /sys/fs/cgroup
>> >> 4 cassandra instances
>> /dev/sdd                           1.5T  802G  688G  54% /data/ssd4
>> /dev/sda                           1.5T  798G  692G  54% /data/ssd1
>> /dev/sdb                           1.5T  681G  810G  46% /data/ssd2
>> /dev/sdc                           1.5T  558G  932G  38% /data/ssd3
>>
>> Cassandra load is about 200GB and the rest of the space is snapshots
>>
>> CPU
>> cass_a@x 127 10:58:47 ~ $ lscpu | grep -E '^Thread|^Core|^Socket|^CPU\('
>> CPU(s):                64
>> Thread(s) per core:    2
>> Core(s) per socket:    16
>> Socket(s):             2
>>
>> *Description of problem:*
>> During repair of the cluster, we are seeing multiple corruptions in the
>> log files on a lot of instances.  There seems to be no pattern to the
>> corruption.  It seems that the repair job is finding all the corrupted
>> files for us.  The repair will hang on the node where the corrupted file is
>> found.  To fix this we remove/rename the datafile and bounce the Cassandra
>> instance.  Our hardware/OS team have stated there is no problem on their
>> side.  I do not believe it the repair causing the corruption.
>>
>> We have maintenance scripts that run every night running compactions and
>> creating snapshots, I decided to turn these off, fix any corruptions we
>> currently had and ran major compaction on all nodes, once this was done we
>> had a "clean" cluster and we left the cluster for a few days.  After the
>> process we noticed one corruption in the cluster, this datafile was created
>> after I turned off the maintenance scripts so my theory of the scripts
>> causing the issue was wrong.  We then kicked off another repair and started
>> to find more corrupt files created after the maintenance script was turned
>> off.
>>
>>
>> So let me give you an example of a corrupted file and maybe someone might
>> be able to work through it with me?
>>
>> When this corrupted file was reported in the log it looks like it was the
>> repair that found it.
>>
>> $ journalctl -u cassmeta-cass_b.service --since "2019-08-07 22:25:00"
>> --until "2019-08-07 22:45:00"
>>
>> Aug 07 22:30:33 cassandra[34611]: INFO  21:30:33 Writing
>> Memtable-compactions_in_progress@830377457(0.008KiB serialized bytes, 1
>> ops, 0%/0% of on/off-heap limit)
>> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Failed creating a merkle
>> tree for [repair #9587a200-b95a-11e9-8920-9f72868b8375 on
>> KeyspaceMetadata/x, (-1476350953672479093,-1474461
>> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Exception in thread
>> Thread[ValidationExecutor:825,1,main]
>> Aug 07 22:30:33 cassandra[34611]: org.apache.cassandra.io.FSReadError:
>> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
>> /x/ssd2/data/KeyspaceMetadata/x-1e453cb0
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:365)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:340)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:382)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:366)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:81)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:169)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>> ~[guava-16.0.jar:na]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:174)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.compaction.LazilyCompactedRow.update(LazilyCompactedRow.java:187)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.repair.Validator.rowHash(Validator.java:201)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.repair.Validator.add(Validator.java:150)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1166)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:76)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:736)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_172]
>> Aug 07 22:30:33 cassandra[34611]: at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> ~[na:1.8.0_172]
>> Aug 07 22:30:33 cassandra[34611]: at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [na:1.8.0_172]
>> Aug 07 22:30:33 cassandra[34611]: at
>> java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]
>> Aug 07 22:30:33 cassandra[34611]: Caused by:
>> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
>> /data/ssd2/data/KeyspaceMetadata/x-x/l
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:216)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:226)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:42)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:352)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: ... 27 common frames omitted
>> Aug 07 22:30:33 cassandra[34611]: Caused by:
>> org.apache.cassandra.io.compress.CorruptBlockException:
>> (/data/ssd2/data/KeyspaceMetadata/x-x/lb-26203-big
>> Aug 07 22:30:33 cassandra[34611]: at
>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:185)
>> ~[apache-cassandra-2.2.13.jar:2.2.13]
>> Aug 07 22:30:33 cassandra[34611]: ... 30 common frames omitted
>> Aug 07 22:30:33 cassandra[34611]: INFO  21:30:33 Not a global repair,
>> will not do anticompaction
>> Aug 07 22:30:33 cassandra[34611]: ERROR 21:30:33 Stopping gossiper
>> Aug 07 22:30:33 cassandra[34611]: WARN  21:30:33 Stopping gossip by
>> operator request
>> Aug 07 22:30:33 cassandra[34611]: INFO  21:30:33 Announcing shutdown
>> Aug 07 22:30:33 cassandra[34611]: INFO  21:30:33 Node /10.2.57.37
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.2.57.37&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=4S7F10IxFntsiwIo-XT-YrkZE8312_yG8jMeOqOBjpE&s=20PLv0KNcUpBbyY1byoboimsLRjbPCLV76xL37jpttQ&e=>
>> state jump to shutdown
>>
>>
>> So I went to the file system to see when this corrupt file was created
>> and it was created on July 30th at 15.55
>>
>> root@x 0 01:14:03 ~ # ls -l
>> /data/ssd2/data/KeyspaceMetadata/x-x/lb-26203-big-Data.db
>> -rw-r--r-- 1 cass_b cass_b 3182243670 Jul 30 15:55
>> /data/ssd2/data/KeyspaceMetadata/x-x/lb-26203-big-Data.db
>>
>>
>>
>> So I checked /var/log/messages for errors around that time
>> The only thing that stands out to me is the message "Cannot perform a
>> full major compaction as repaired and unrepaired sstables cannot be
>> compacted together", I'm not sure if this would be an issue though and
>> cause corruption.
>>
>> Jul 30 15:55:06 x systemd: Created slice User Slice of root.
>> Jul 30 15:55:06 x systemd: Started Session c165280 of user root.
>> Jul 30 15:55:06 x audispd: node=x. type=USER_START
>> msg=audit(1564498506.021:457933): pid=17533 uid=0 auid=4294967295
>> ses=4294967295 msg='op=PAM:session_open
>> grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_tty_audit,pam_systemd,pam_unix
>> acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
>> Jul 30 15:55:06 x systemd: Removed slice User Slice of root.
>> Jul 30 15:55:14 x tag_audit_log: type=USER_CMD
>> msg=audit(1564498506.013:457932): pid=17533 uid=509 auid=4294967295
>> ses=4294967295 msg='cwd="/"
>> cmd=2F7573722F7362696E2F69706D692D73656E736F7273202D2D71756965742D6361636865202D2D7364722D63616368652D7265637265617465202D2D696E746572707265742D6F656D2D64617461202D2D6F75747075742D73656E736F722D7374617465202D2D69676E6F72652D6E6F742D617661696C61626C652D73656E736F7273202D2D6F75747075742D73656E736F722D7468726573686F6C6473
>> terminal=? res=success'
>> Jul 30 15:55:14 x tag_audit_log: type=USER_START
>> msg=audit(1564498506.021:457933): pid=17533 uid=0 auid=4294967295
>> ses=4294967295 msg='op=PAM:session_open
>> grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_tty_audit,pam_systemd,pam_unix
>> acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
>> Jul 30 15:55:19 x cassandra: INFO  14:55:19 Writing
>> Memtable-compactions_in_progress@1462227999(0.008KiB serialized bytes, 1
>> ops, 0%/0% of on/off-heap limit)
>> Jul 30 15:55:19 x cassandra: INFO  14:55:19 Cannot perform a full major
>> compaction as repaired and unrepaired sstables cannot be compacted
>> together. These two set of sstables will be compacted separately.
>> Jul 30 15:55:19 x cassandra: INFO  14:55:19 Writing
>> Memtable-compactions_in_progress@1198535528(1.002KiB serialized bytes,
>> 57 ops, 0%/0% of on/off-heap limit)
>> Jul 30 15:55:20 x cassandra: INFO  14:55:20 Writing
>> Memtable-compactions_in_progress@2039409834(0.008KiB serialized bytes, 1
>> ops, 0%/0% of on/off-heap limit)
>> Jul 30 15:55:24 x audispd: node=x. type=USER_LOGOUT
>> msg=audit(1564498524.409:457934): pid=46620 uid=0 auid=464400029 ses=2747
>> msg='op=login id=464400029 exe="/usr/sbin/sshd" hostname=? addr=?
>> terminal=/dev/pts/0 res=success'
>> Jul 30 15:55:24 x audispd: node=x. type=USER_LOGOUT
>> msg=audit(1564498524.409:457935): pid=4878 uid=0 auid=464400029 ses=2749
>> msg='op=login id=464400029 exe="/usr/sbin/sshd" hostname=? addr=?
>> terminal=/dev/pts/1 res=success'
>>
>> Jul 30 15:55:57 x systemd: Created slice User Slice of root.
>> Jul 30 15:55:57 x systemd: Started Session c165288 of user root.
>> Jul 30 15:55:57 x audispd: node=x. type=USER_START
>> msg=audit(1564498557.294:457958): pid=19687 uid=0 auid=4294967295
>> ses=4294967295 msg='op=PAM:session_open
>> grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_tty_audit,pam_systemd,pam_unix
>> acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
>> Jul 30 15:55:57 x audispd: node=x. type=USER_START
>> msg=audit(1564498557.298:457959): pid=19690 uid=0 auid=4294967295
>> ses=4294967295 msg='op=PAM:session_open
>> grantors=pam_keyinit,pam_systemd,pam_keyinit,pam_limits,pam_unix
>> acct="cass_b" exe="/usr/sbin/runuser" hostname=? addr=? terminal=?
>> res=success'
>> Jul 30 15:55:58 x systemd: Removed slice User Slice of root.
>> Jul 30 15:56:02 x cassandra: INFO  14:56:02 Writing
>> Memtable-compactions_in_progress@1532791194(0.008KiB serialized bytes, 1
>> ops, 0%/0% of on/off-heap limit)
>> Jul 30 15:56:02 x cassandra: INFO  14:56:02 Cannot perform a full major
>> compaction as repaired and unrepaired sstables cannot be compacted
>> together. These two set of sstables will be compacted separately.
>> Jul 30 15:56:02 x cassandra: INFO  14:56:02 Writing
>> Memtable-compactions_in_progress@1455399453(0.281KiB serialized bytes,
>> 16 ops, 0%/0% of on/off-heap limit)
>> Jul 30 15:56:04 x tag_audit_log: type=USER_CMD
>> msg=audit(1564498555.190:457951): pid=19294 uid=509 auid=4294967295
>> ses=4294967295 msg='cwd="/"
>> cmd=72756E75736572202D73202F62696E2F62617368202D6C20636173735F62202D632063617373616E6472612D6D6574612F63617373616E6472612F62696E2F6E6F6465746F6F6C2074707374617473
>> terminal=? res=success'
>>
>>
>>
>> We have checked a number of other things like NTP setting etc but nothing
>> is telling us what could cause so many corruptions across the entire
>> cluster.
>> Things were healthy with this cluster for months, the only thing I can
>> think is that we started loading data from a load of 20GB per instance up
>> to 200GB where it sits now, maybe this just highlighted the issue.
>>
>>
>>
>> Compaction and Compression on Keyspace CL's [mixture]
>> All CF's are using compression.
>>
>> AND compaction = {'min_threshold': '4', 'class':
>> 'org.apache.cassandra.db.compaction.*SizeTieredCompactionStrategy*',
>> 'max_threshold': '32'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>> AND compaction = {'min_threshold': '4', 'class':
>> 'org.apache.cassandra.db.compaction.*SizeTieredCompactionStrategy*',
>> 'max_threshold': '32'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*LZ4Compressor*'}
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>> *LeveledCompactionStrategy*'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>> --We are also using internode network compression:
>> internode_compression: all
>>
>>
>>
>> Does anyone have any idea what I should check next?
>> Our next theory is that there may be an issue with Checksum, but I'm not
>> sure where to go with this.
>>
>>
>>
>> Any help would be very much appreciated before I lose the last bit of
>> hair I have on my head.
>>
>>
>>
>> Kind Regards,
>>
>> Phil
>>
>>
>>
>> On Wed, 7 Aug 2019 at 20:51, Nitan Kainth <nitankainth@gmail.com> wrote:
>>
>> Repair during upgrade have caused corruption too.
>>
>>
>>
>> Also, dropping and adding columns with same name but different type
>>
>>
>>
>> Regards,
>>
>> Nitan
>>
>> Cell: 510 449 9629
>>
>>
>> On Aug 7, 2019, at 2:42 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>>
>> Is compression enabled?
>>
>>
>>
>> If not, bit flips on disk can corrupt data files and reads + repair may
>> send that corruption to other hosts in the cluster
>>
>>
>> On Aug 7, 2019, at 3:46 AM, Philip Ó Condúin <philipoconduin@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am currently experiencing multiple datafile corruptions across most
>> nodes in my cluster, there seems to be no pattern to the corruption.  I'm
>> starting to think it might be a bug, we're using Cassandra 2.2.13.
>>
>>
>>
>> Without going into detail about the issue I just want to confirm
>> something.
>>
>>
>>
>> Can someone share with me a list of scenarios that would cause corruption?
>>
>>
>>
>> 1. OS failure
>>
>> 2. Cassandra disturbed during the writing
>>
>>
>>
>> etc etc.
>>
>>
>>
>> I need to investigate each scenario and don't want to leave any out.
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Phil
>>
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Phil
>>
>
>
> --
> Regards,
> Phil
>

Mime
View raw message