Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 086FB200D2B for ; Thu, 2 Nov 2017 22:40:28 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 06E4D160BE5; Thu, 2 Nov 2017 21:40:28 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CE3781609EB for ; Thu, 2 Nov 2017 22:40:26 +0100 (CET) Received: (qmail 91643 invoked by uid 500); 2 Nov 2017 21:40:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 91633 invoked by uid 99); 2 Nov 2017 21:40:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Nov 2017 21:40:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 27DCA18060F for ; Thu, 2 Nov 2017 21:40:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.479 X-Spam-Level: ** X-Spam-Status: No, score=2.479 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=instaclustr-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id YKAlywzJLIeU for ; Thu, 2 Nov 2017 21:40:20 +0000 (UTC) Received: from mail-yw0-f179.google.com (mail-yw0-f179.google.com [209.85.161.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 06EE25FDE8 for ; Thu, 2 Nov 2017 21:40:19 +0000 (UTC) Received: by mail-yw0-f179.google.com with SMTP id t71so855267ywc.3 for ; Thu, 02 Nov 2017 14:40:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=instaclustr-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=xZWAy/UyTkW25kjisa3E5E83vOd8WvTX1ieYJWKBN0E=; b=kTzhdM1+eljcOR23TLEITXb5g4MPMP3KskvOO4yk9dKxqQvFCVYz7b6zQtxjGPddyz Sc85mWv02cg+OEAhp7xswbE6w6GylMo3jqT7/RD/8v+0ZUKKMR7cT651ExruCryEkYp9 4UA7VB61usWgNuhKlEpt+3bHH6GzgLRHFRGGMwmZHJUjIkl5a/0ij2qDojGQKGZN6EEQ jmZFYXRVMPwQoDbJHTXvBowp/jBN8LFlhPVRCGT6xdn+Tfpg7BRYDyIp+qa4x6cag+wr d+PKpIOU+xPQazp6T8fVnUf0xyXxwDl0hEx3HkeSIwsKuObhuJ4KloBlK3tax7MySe+c Ji8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=xZWAy/UyTkW25kjisa3E5E83vOd8WvTX1ieYJWKBN0E=; b=LLwUXcuK/THhnNTXw+oY/bAX2ncBn2rhEZ9IWIWINO7wBfpOgRlCXoVnVVfLogibai c/ZxY2SniJaABOXId+ReZAade8tGhK5xABLBtRWLJtAOPT3LACe1GuagPSxB0S1pjZpn LCDD0Fz4wo99rEhUmbhGdq11y3fCBXrrO1s+eJUFc/YitXktmzobQeF6OxXKBxKz2FCk ezFcocfd3UYz29P4ffoG7le/5mWsLc19k8CYegia64o9qMSQv/1r6yAWgyi6mGrr9NGZ Lwi3oNogZ/fYm2KsIaPFufyoXT+5X4B5il1/xP8BWE+lOTOo9IE020I374JtIKDcXKoe QfVA== X-Gm-Message-State: AMCzsaV/kzrSJnXnt7zsWVvUbCR1hRYmdevjedxjOEEsSLRFzqqMcO6y 7Lr4iS9TBU2b1MaESQEjz+bIB2dolM6cjqtx1i3zcRMQ X-Google-Smtp-Source: ABhQp+QOiPl0gHjwhnKMO1VqBuFGcioCm4kdrB+wC/+9BNvVPQJ+yWk83eg49hvpFaLku4XHu755qXYgmxTIWvxY7t8= X-Received: by 10.129.135.65 with SMTP id x62mr3355501ywf.267.1509658819412; Thu, 02 Nov 2017 14:40:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.242.132 with HTTP; Thu, 2 Nov 2017 14:40:18 -0700 (PDT) Received: by 10.13.242.132 with HTTP; Thu, 2 Nov 2017 14:40:18 -0700 (PDT) In-Reply-To: References: From: kurt greaves Date: Thu, 2 Nov 2017 21:40:18 +0000 Message-ID: Subject: Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version To: User Content-Type: multipart/alternative; boundary="001a114f62d0a9f33d055d06d589" archived-at: Thu, 02 Nov 2017 21:40:28 -0000 --001a114f62d0a9f33d055d06d589 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Well, pretty sure they still are. at least the mutation one is. but you should really use the dedicated metrics for this. On 3 Nov. 2017 01:38, "Anumod Mullachery" wrote: > thanks .. > > so the dropped hints & messages are not captured in cassandra logs, post > 3.x vs 2.x. > > -Anumod > > > > On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves wrote= : > >> You can get dropped message statistics over JMX. for example nodetool >> tpstats has a counter for dropped hints from startup. that would be the >> preferred method for tracking this info, rather than parsing logs >> >> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" >> wrote: >> >> >> Hi All, >> >> In cassandra v 2.1.15 , I'm able to pull the hints drop and dropped >> messages from cassandra.log as below- >> >> >> dropped hints--> >> >> "/opt/xcal/apps/cassandra/logs/cassandra.log >> >> --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints, >> because node is down past configured hint window." >> >> Dropped messages --> >> >> "/opt/xcal/apps/cassandra/logs/cassandra.log >> >> --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms" >> >> >> But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v >> 2.1.15. >> >> I'm not able to get any info from the cassandra logs for dropped hints & >> dropped messages. ( I was testing this on Test cluster by stopping the >> cassandra services on one of the node , where the hints window is set to= 05 >> Mins and kept the node down for 1 hour , and monitoring the cassandra lo= gs, >> but nothing generated on dropped hints on any other nodes in the cluster= ). >> >> The only message I found is on the down node is that - "Paused hints >> dispatch". >> >> Can some one put some light on this issue on 3.10 , is there any changes >> in the hints & dropped messages , logs or the process. >> >> thanks in advance, >> >> regards, >> >> Anumod. >> >> >> > --001a114f62d0a9f33d055d06d589 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Well, pretty sure they still are. at least the mutation o= ne is. but you should really use the dedicated metrics for this.

On 3 Nov. 2017 01:38, = "Anumod Mullachery" <anumodmullacheryc@gmail.com> wrote:
thanks ..

so the dropped hints & messages are not captured in cassandra logs,= post 3.x vs 2.x.

-Anumod



On Wed, Nov 1,= 2017 at 4:50 PM, kurt greaves <kurt@instaclustr.com> wro= te:
You can get dropped= message statistics over JMX. for example nodetool tpstats has a counter fo= r dropped hints from startup. that would be the preferred method for tracki= ng this info, rather than parsing logs

On 2 Nov. 2017 6:24 am, "Anumod Mu= llachery" <anumodmullacheryc@gmail.com> wrote:

Hi All,

In cassandra=C2= =A0 v 2.1.15 , I'm able to pull the hints drop and dropped messages fro= m cassandra.log as below-


dropped hints-->

= = "/opt/xcal/apps/cassandr= a/logs/cassandra.log=C2=A0 --> = HintedHandoffMetrics.java:79 - /96.115.91.69= has 5 dropped hints, because node is down past conf= igured hint wind= ow."

= Dropped messages -->

=
= = &quo= t;/opt/xcal/apps/cassandra/logs/cassandra.log = --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"=


B= ut in Cassandra V 3.10 , I'm not able to get the cassandra logs = as v 2.1.15.

I'm not able to get any info from= the cassandra logs for dropped hints & dropped messages. ( I was testi= ng this on Test cluster by stopping the cassandra services on one of the no= de , where the hints window is set to 05 Mins and kept the node down for 1 = hour , and monitoring the cassandra logs, but nothing generated on dropped = hints on any other nodes in the cluster ).

The onl= y message I found is on the down node is that - "Pause= d hints di= spatch".

Can some one put some light on this issue on 3.10 , is there any ch= anges in the hints & dropped messages , logs or the process.
=

thanks in ad= vance,

regards,

Anumod.



--001a114f62d0a9f33d055d06d589--