From user-return-64646-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Thu Oct 24 20:21:39 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 032E218065D for ; Thu, 24 Oct 2019 22:21:38 +0200 (CEST) Received: (qmail 45685 invoked by uid 500); 24 Oct 2019 20:21:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 45675 invoked by uid 99); 24 Oct 2019 20:21:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Oct 2019 20:21:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D314D1A412E for ; Thu, 24 Oct 2019 20:21:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 3ZMbvC6OJi46 for ; Thu, 24 Oct 2019 20:21:31 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.174; helo=mail-lj1-f174.google.com; envelope-from=lapostadisergio@gmail.com; receiver= Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 4A46CBC57A for ; Thu, 24 Oct 2019 19:57:43 +0000 (UTC) Received: by mail-lj1-f174.google.com with SMTP id j19so38724lja.1 for ; Thu, 24 Oct 2019 12:57:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ryhdLRqJuMnDlierwjgIuEnbZXRQFNmIHOTgHtL8VQk=; b=aY3mCMO9GtseGYxuGTrdZS7WfT51R3ZCJ+2xZkzh0t1e+0rqBl58N7ElI0Dm+yRJ2R tVhPMsLWiOvNSpgZrrUTmdSsGhL7I/Uq5kDYIlSIjWrULlLN50eWMgAT8R5oIn2zxioc I7aWZ/adWxQcZP2EdTIeH7PsgHSalAqVqSLnHnXPrs6xQDLzmEgQaMYCzuj51U9CdwBd CaE+WcmHsBioWOMz6nvCLzM6KL6BCMOaCWrKCkt/R65eqMIX8ieX2J5rFQMCZXhptKaa 956yls6wYWzDWPsOsosgk5/Kroio6TFI8TsLdj+0zlsxEJjMMKQHV/LxjHMHYJ28lBpf S3AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ryhdLRqJuMnDlierwjgIuEnbZXRQFNmIHOTgHtL8VQk=; b=EAHfRII0BnhJaGglimyuMItFuUXXIxvC+wrR+SRhMZ1Ps4N4oRgcKiFyKCcsR2kTfX k8VBKgY2oCGXdVQNNQJV40A66rhz4KsCTY2IjKe3P2zU5aNtzfCx0j6oXR7aqLy+0PQv /6yYUMuY5ejTYgOAEcbDVIe32/gPoIsYpGYWxyJWQpKq0jWhYXbx/83zU2aqKHIgIT9O nhYLky0CINJNzTtz/xsckYVlsM9jYG5OUsAaDy365kDy23i+C2Mh07xWO9ltbDqdM9DM fZCKuHOOMZwCunwSMboA+G3F1Qyeh1cgLu+WbJiiFCjK0YUFol7GA6FQNkx57csN8Tie 2p0A== X-Gm-Message-State: APjAAAU+RLe8N5IRvvTAoiVPFOru3+DP7jk9E8TnWXg29fKyZZJaNsNf SHfWr+Va9c3Ld1JU71uxzt4KuDKAmU9SgPHEEuZdaA== X-Google-Smtp-Source: APXvYqyK5OF18ewju/6vF8vut6AxQy0fyynKUKC7NujlZ7XNdMNc5pQevGFXMZZWxcWMUez292S8LUgW9xs18+I13nE= X-Received: by 2002:a2e:9b02:: with SMTP id u2mr27427069lji.18.1571947055602; Thu, 24 Oct 2019 12:57:35 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Sergio Date: Thu, 24 Oct 2019 12:57:22 -0700 Message-ID: Subject: Re: Repair Issues To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000dae13e0595ad71f2" --000000000000dae13e0595ad71f2 Content-Type: text/plain; charset="UTF-8" Are you using Cassandra reaper? On Thu, Oct 24, 2019, 12:31 PM Ben Mills wrote: > Greetings, > > Inherited a small Cassandra cluster with some repair issues and need some > advice on recommended next steps. Apologies in advance for a long email. > > Issue: > > Intermittent repair failures on two non-system keyspaces. > > - platform_users > - platform_management > > Repair Type: > > Full, parallel repairs are run on each of the three nodes every five days. > > Repair command output for a typical failure: > > [2019-10-18 00:22:09,109] Starting repair command #46, repairing keyspace > platform_users with repair options (parallelism: parallel, primary range: > false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: > [], hosts: [], # of ranges: 12) > [2019-10-18 00:22:09,242] Repair session > 5282be70-f13d-11e9-9b4e-7f6db768ba9a for range > [(-1890954128429545684,2847510199483651721], > (8249813014782655320,-8746483007209345011], > (4299912178579297893,6811748355903297393], > (-8746483007209345011,-8628999431140554276], > (-5865769407232506956,-4746990901966533744], > (-4470950459111056725,-1890954128429545684], > (4001531392883953257,4299912178579297893], > (6811748355903297393,6878104809564599690], > (6878104809564599690,8249813014782655320], > (-4746990901966533744,-4470950459111056725], > (-8628999431140554276,-5865769407232506956], > (2847510199483651721,4001531392883953257]] failed with error [repair > #5282be70-f13d-11e9-9b4e-7f6db768ba9a on platform_users/access_tokens_v2, > [(-1890954128429545684,2847510199483651721], > (8249813014782655320,-8746483007209345011], > (4299912178579297893,6811748355903297393], > (-8746483007209345011,-8628999431140554276], > (-5865769407232506956,-4746990901966533744], > (-4470950459111056725,-1890954128429545684], > (4001531392883953257,4299912178579297893], > (6811748355903297393,6878104809564599690], > (6878104809564599690,8249813014782655320], > (-4746990901966533744,-4470950459111056725], > (-8628999431140554276,-5865769407232506956], > (2847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x > (progress: 26%) > [2019-10-18 00:22:09,246] Some repair failed > [2019-10-18 00:22:09,248] Repair command #46 finished in 0 seconds > > Additional Notes: > > Repairs encounter above failures more often than not. Sometimes on one > node only, though occasionally on two. Sometimes just one of the two > keyspaces, sometimes both. Apparently the previous repair schedule for > this cluster included incremental repairs (script alternated between > incremental and full repairs). After reading this TLP article: > > > https://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html > > the repair script was replaced with cassandra-reaper (v1.4.0), which was > run with its default configs. Reaper was fine but only obscured the ongoing > issues (it did not resolve them) and complicated the debugging process and > so was then removed. The current repair schedule is as described above > under Repair Type. > > Attempts at Resolution: > > (1) nodetool scrub was attempted on the offending keyspaces/tables to no > effect. > > (2) sstablescrub has not been attempted due to the current design of the > Docker image that runs Cassandra in each Kubernetes pod - i.e. there is no > way to stop the server to run this utility without killing the only pid > running in the container. > > Related Error: > > Not sure if this is related, though sometimes, when either: > > (a) Running nodetool snapshot, or > (b) Rolling a pod that runs a Cassandra node, which calls nodetool drain > prior shutdown, > > the following error is thrown: > > -- StackTrace -- > java.lang.RuntimeException: Last written key > DecoratedKey(10df3ba1-6eb2-4c8e-bddd-c0c7af586bda, > 10df3ba16eb24c8ebdddc0c7af586bda) >= current key > DecoratedKey(00000000-0000-0000-0000-000000000000, > 17343121887f480c9ba87c0e32206b74) writing into > /cassandra_data/data/platform_management/device_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx/mb-45-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:114) > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153) > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:441) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:477) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:363) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Here are some details on the environment and configs in the event that > something is relevant. > > Environment: Kubernetes > Environment Config: Stateful set of 3 replicas > Storage: Persistent Volumes > Storage Class: SSD > Node OS: Container-Optimized OS > Container OS: Ubuntu 16.04.3 LTS > > Version: Cassandra 3.7 > Data Centers: 1 > Racks: 3 (one per zone) > Nodes: 3 > Tokens: 4 > Replication Factor: 3 > Replication Strategy: NetworkTopologyStrategy (all keyspaces) > Compaction Strategy: STCS (all tables) > Read/Write Requirements: Blend of both > Data Load: <1GB per node > gc_grace_seconds: default (10 days - all tables) > > Memory: 4Gi per node > CPU: 3.5 per node (3500m) > > Java Version: 1.8.0_144 > > Heap Settings: > > -XX:+UnlockExperimentalVMOptions > -XX:+UseCGroupMemoryLimitForHeap > -XX:MaxRAMFraction=2 > > GC Settings: (CMS) > > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > -XX:CMSWaitDuration=30000 > -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways > > Any ideas are much appreciated. > --000000000000dae13e0595ad71f2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Are you using Cassandra reaper?

On Thu, Oct 24, 2019, 12:3= 1 PM Ben Mills <ben@bitbrew.com&g= t; wrote:
Greetings,

Inherited a small Cassand= ra cluster with some repair issues and need some advice on recommended next= steps. Apologies in advance for a long email.

Issue:

Intermi= ttent repair failures on two non-system keyspaces.

- platform_users<= br>- platform_management

Repair Type:

Full, parallel repairs = are run on each of the three nodes every five days.

Repair command o= utput for a typical failure:

[2019-10-18 00:22:09,109] Starting repa= ir command #46, repairing keyspace platform_users with repair options (para= llelism: parallel, primary range: false, incremental: false, job threads: 1= , ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 12)
[2019= -10-18 00:22:09,242] Repair session 5282be70-f13d-11e9-9b4e-7f6db768ba9a fo= r range [(-1890954128429545684,2847510199483651721], (8249813014782655320,-= 8746483007209345011], (4299912178579297893,6811748355903297393], (-87464830= 07209345011,-8628999431140554276], (-5865769407232506956,-47469909019665337= 44], (-4470950459111056725,-1890954128429545684], (4001531392883953257,4299= 912178579297893], (6811748355903297393,6878104809564599690], (6878104809564= 599690,8249813014782655320], (-4746990901966533744,-4470950459111056725], (= -8628999431140554276,-5865769407232506956], (2847510199483651721,4001531392= 883953257]] failed with error [repair #5282be70-f13d-11e9-9b4e-7f6db768ba9a= on platform_users/access_tokens_v2, [(-1890954128429545684,284751019948365= 1721], (8249813014782655320,-8746483007209345011], (4299912178579297893,681= 1748355903297393], (-8746483007209345011,-8628999431140554276], (-586576940= 7232506956,-4746990901966533744], (-4470950459111056725,-189095412842954568= 4], (4001531392883953257,4299912178579297893], (6811748355903297393,6878104= 809564599690], (6878104809564599690,8249813014782655320], (-474699090196653= 3744,-4470950459111056725], (-8628999431140554276,-5865769407232506956], (2= 847510199483651721,4001531392883953257]]] Validation failed in /10.x.x.x (p= rogress: 26%)
[2019-10-18 00:22:09,246] Some repair failed
[2019-10-1= 8 00:22:09,248] Repair command #46 finished in 0 seconds

Additional = Notes:

Repairs encounter above failures more often than not. Sometim= es on one node only, though occasionally on two. Sometimes just one of the = two keyspaces, sometimes both. Apparen= tly the previous repair schedule for this cluster inclu= ded incremental repairs (script alternated between incremental and full rep= airs). After reading this TLP article:

https://thelastpickle.com/blog/2017/12/14/should-= you-use-incremental-repair.html

the repair script was replaced w= ith cassandra-reaper (v1.4.0), which was run with its default configs. Reap= er was fine but only obscured the ongoing issues (it did not resolve them) = and complicated the debugging process and so was then removed. The current = repair schedule is as described above under Repair Type.

Attempts at= Resolution:

(1) nodetool scrub was attempted on the offending keysp= aces/tables to no effect.

(2) sstablescrub has not been attempted du= e to the current design of the Docker image that runs Cassandra in each Kub= ernetes pod - i.e. there is no way to stop the server to run this utility w= ithout killing the only pid running in the container.

Related Error:=

Not sure if this is related, though sometimes, when either:

= (a) Running nodetool snapshot, or
(= b) Rolling=C2=A0a pod that runs a Cass= andra node, which calls nodetool drain prior shutdown,

the following error is thrown:

-- StackTrace= --
java.lang.RuntimeException: Last written key DecoratedKey(10df3ba1-6= eb2-4c8e-bddd-c0c7af586bda, 10df3ba16eb24c8ebdddc0c7af586bda) >=3D curre= nt key DecoratedKey(00000000-0000-0000-0000-000000000000, 17343121887f480c9= ba87c0e32206b74) writing into /cassandra_data/data/platform_management/devi= ce_by_tenant_v2-e91529202ccf11e7ab96d5693708c583/.device_by_tenant_tags_idx= /mb-45-big-Data.db
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apac= he.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWrit= er.java:114)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cas= sandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.io.ss= table.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.Memtab= le$FlushRunnable.writeSortedContents(Memtable.java:441)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.Memtable$FlushRunnab= le.call(Memtable.java:477)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at = org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:363)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.FutureTas= k.run(FutureTask.java:266)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 149)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.T= hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:748)
Here are some details on the environment and configs in the event that som= ething is relevant.

Environment: Kubernetes
Environment Config: S= tateful set of 3 replicas
Storage: Persistent Volumes
Storage Class: = SSD
Node OS: Container-Optimized OS
Container OS: Ubuntu 16.04.3 LTS<= br>
Version: Cassandra 3.7
Data Centers: 1
Racks: 3 (one per zone)=
Nodes: 3
Tokens: 4
Replication Factor: 3
Replication Strategy:= NetworkTopologyStrategy (all keyspaces)
Compaction Strategy: STCS (all = tables)
Read/Write Requirements: Blend of both
Data Load: <1GB per= node
gc_grace_seconds: default (10 days - all tables)

Memory: 4G= i per node
CPU: 3.5 per node (3500m)

Java Version: 1.8.0_144
<= br>Heap Settings:

-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroup= MemoryLimitForHeap
-XX:MaxRAMFraction=3D2

GC Settings: (CMS)
<= br>-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEna= bled
-XX:SurvivorRatio=3D8
-XX:MaxTenuringThreshold=3D1
-XX:CMSIni= tiatingOccupancyFraction=3D75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:= CMSWaitDuration=3D30000
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSEd= enChunksRecordAlways
Any ideas are mu= ch appreciated.=C2=A0
--000000000000dae13e0595ad71f2--