From user-return-65154-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Feb 7 01:56:25 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D455818064E for ; Fri, 7 Feb 2020 02:56:24 +0100 (CET) Received: (qmail 93742 invoked by uid 500); 7 Feb 2020 01:56:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 93732 invoked by uid 99); 7 Feb 2020 01:56:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Feb 2020 01:56:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F2F07181394 for ; Fri, 7 Feb 2020 01:56:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.624 X-Spam-Level: * X-Spam-Status: No, score=1.624 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HK_RANDOM_ENVFROM=0.626, HK_RANDOM_FROM=0.999, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 1omOC-EaK5cv for ; Fri, 7 Feb 2020 01:56:19 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.51; helo=mail-wm1-f51.google.com; envelope-from=flightctlr@gmail.com; receiver= Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id C13B3BB806 for ; Fri, 7 Feb 2020 01:56:18 +0000 (UTC) Received: by mail-wm1-f51.google.com with SMTP id q9so897507wmj.5 for ; Thu, 06 Feb 2020 17:56:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=c06RjApDsFQ8chHUcVgsa6eoi3WxUxOTuF7o4SJIoMk=; b=RXmKFcQZsctnSpnTSRICv4ppBHljThPGFw+D9h4Z8wC/A0sfKUk6+s0tLMAz3GQU6Z 62WMjpwBnKkjILamRfEidiZklQ4icaUlVADcwSwU6/ndGG4eeWHCFkNH9MC+Mgqnl6ww mJ3mHWo/jLMAVgbfsjqRAeyFY8hszshtmCY+Ood8bCfviRbOAfrfXtFjyGL9m5EyduwM I9i7COiEzvUHqhlwfaaJZhQ4BVbe3/MOe2Qv3C72tRtiM4H1GcNwOMzsRbvykxF7E/ex dkOpEgK2+DFhIzhtftuAcmZd2hZcznRwecl0AazacDYxhapheGFA5liSnGUTUVsl2fUM AcLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=c06RjApDsFQ8chHUcVgsa6eoi3WxUxOTuF7o4SJIoMk=; b=JHl48Q6UYaHLlepp7c1rDHLxuIv+opgBf5OcveaOKNOlaxC7ea5f2qec2YXbHS3jD/ Mp1qZfB0Lvz6itZbB4hcJ3L5lCoAglbXGuGNFzZeJmhxX3xlLC1ItaDoEA7Wp4W1D7jb 6Rfo/2OglTgEpT69vMAUTgKi50+Qqva0SQyIoVXGMio5r87RJoAZf7BV2VJPT9n1mWMi hrK9caAsDfxtZz2Zfq24dRsy2f5uQ4wn0H4jqfBmpDhCX2X+sta/neWMjS6rJmlGsq5z //PEA6bArlXNZsxU3iohzXl1AkcuKXYPdFRz5YPaYrXB0spN32RpIKv1ql+VlT+1F9d4 ztzA== X-Gm-Message-State: APjAAAXPRCoXBQumNVnZQI5YhlpPv2DPiNH9tC5Okmcjvo/u8bIx5FCf xGiM1/Umvh5bak86xJZJmeHvwyJqCFG/FOTaxfiTug== X-Google-Smtp-Source: APXvYqx4RcBV4cHIfqhMIDXWOsdk3piL+yC9rky43Hkwm/TkxHWmrKAFFnKw6SeHoN+cUUPhwbpiwUthu+8iQzSoJUo= X-Received: by 2002:a1c:7717:: with SMTP id t23mr976853wmi.17.1581040572092; Thu, 06 Feb 2020 17:56:12 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Erick Ramirez Date: Fri, 7 Feb 2020 12:56:01 +1100 Message-ID: Subject: Re: Nodes becoming unresponsive To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000acd986059df2b1eb" --000000000000acd986059df2b1eb Content-Type: text/plain; charset="UTF-8" > > I tried to debug more and could see using top that Command is > MutationStage in top output , Any clue we get from this ? > That just means there's lots of writes hitting your cluster. Without the thread dump, it would be difficult to know if the threads are blocked by futex_wait or whatever else is going on. Cheers! --000000000000acd986059df2b1eb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I tried to debug = more and could see using top that Command is MutationStage in top output , = Any clue we get from this ?

=C2=A0That just means there's lots of writes hitting your cluster. W= ithout the thread dump, it would be difficult to know if the threads are bl= ocked by futex_wait or whatever else is going on. Cheers!
--000000000000acd986059df2b1eb--