Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6EC99200B61 for ; Tue, 9 Aug 2016 21:55:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6D54D160AB0; Tue, 9 Aug 2016 19:55:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B6E78160A6B for ; Tue, 9 Aug 2016 21:55:25 +0200 (CEST) Received: (qmail 77628 invoked by uid 500); 9 Aug 2016 19:55:24 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 77605 invoked by uid 99); 9 Aug 2016 19:55:24 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2016 19:55:24 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id D92082C02B5 for ; Tue, 9 Aug 2016 19:55:22 +0000 (UTC) Date: Tue, 9 Aug 2016 19:55:22 +0000 (UTC) From: "Nate McCall (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Reopened] (CASSANDRA-11363) High Blocked NTR When Connecting MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 09 Aug 2016 19:55:26 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-11363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate McCall reopened CASSANDRA-11363: ------------------------------------- Reproduced In: 3.0.3, 2.1.13, 2.1.12 (was: 2.1.12, 2.1.13, 3.0.3) Sorry, but I am re-opening this. With have a potential explanation with workaround and I still see it with varying degrees on *every* 2.x+ cluster we have come in contact with recently. > High Blocked NTR When Connecting > -------------------------------- > > Key: CASSANDRA-11363 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11363 > Project: Cassandra > Issue Type: Bug > Components: Coordination > Reporter: Russell Bradberry > Assignee: Paulo Motta > Attachments: cassandra-102-cms.stack, cassandra-102-g1gc.stack, max_queued_ntr_property.txt, thread-queue-2.1.txt > > > When upgrading from 2.1.9 to 2.1.13, we are witnessing an issue where the machine load increases to very high levels (> 120 on an 8 core machine) and native transport requests get blocked in tpstats. > I was able to reproduce this in both CMS and G1GC as well as on JVM 7 and 8. > The issue does not seem to affect the nodes running 2.1.9. > The issue seems to coincide with the number of connections OR the number of total requests being processed at a given time (as the latter increases with the former in our system) > Currently there is between 600 and 800 client connections on each machine and each machine is handling roughly 2000-3000 client requests per second. > Disabling the binary protocol fixes the issue for this node but isn't a viable option cluster-wide. > Here is the output from tpstats: > {code} > Pool Name Active Pending Completed Blocked All time blocked > MutationStage 0 8 8387821 0 0 > ReadStage 0 0 355860 0 0 > RequestResponseStage 0 7 2532457 0 0 > ReadRepairStage 0 0 150 0 0 > CounterMutationStage 32 104 897560 0 0 > MiscStage 0 0 0 0 0 > HintedHandoff 0 0 65 0 0 > GossipStage 0 0 2338 0 0 > CacheCleanupExecutor 0 0 0 0 0 > InternalResponseStage 0 0 0 0 0 > CommitLogArchiver 0 0 0 0 0 > CompactionExecutor 2 190 474 0 0 > ValidationExecutor 0 0 0 0 0 > MigrationStage 0 0 10 0 0 > AntiEntropyStage 0 0 0 0 0 > PendingRangeCalculator 0 0 310 0 0 > Sampler 0 0 0 0 0 > MemtableFlushWriter 1 10 94 0 0 > MemtablePostFlush 1 34 257 0 0 > MemtableReclaimMemory 0 0 94 0 0 > Native-Transport-Requests 128 156 387957 16 278451 > Message type Dropped > READ 0 > RANGE_SLICE 0 > _TRACE 0 > MUTATION 0 > COUNTER_MUTATION 0 > BINARY 0 > REQUEST_RESPONSE 0 > PAGED_RANGE 0 > READ_REPAIR 0 > {code} > Attached is the jstack output for both CMS and G1GC. > Flight recordings are here: > https://s3.amazonaws.com/simple-logs/cassandra-102-cms.jfr > https://s3.amazonaws.com/simple-logs/cassandra-102-g1gc.jfr > It is interesting to note that while the flight recording was taking place, the load on the machine went back to healthy, and when the flight recording finished the load went back to > 100. -- This message was sent by Atlassian JIRA (v6.3.4#6332)