Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 28 Dec 2016 16:32:58 +0000 (UTC)
From: "Corentin Chary (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13027740.1481622998000.607188.1482942778363@Atlassian.JIRA>
In-Reply-To: <JIRA.13027740.1481622998000@Atlassian.JIRA>
References: <JIRA.13027740.1481622998000@Atlassian.JIRA> <JIRA.13027740.1481622998623@arcas>
Subject: [jira] [Commented] (CASSANDRA-13039) Mutation time mostly spent in
 LinkedBlockingQueue.put()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 28 Dec 2016 16:33:05 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15783227#comment-15783227 ] 

Corentin Chary commented on CASSANDRA-13039:
--------------------------------------------

That would make sense when using QUORUM (because you are waiting for these replicated mutation).
If you use ONE it's the opposite. I'm still trying to build a cassandra stress profile that can easilly reproduce that.


> Mutation time mostly spent in LinkedBlockingQueue.put()
> -------------------------------------------------------
>
>                 Key: CASSANDRA-13039
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13039
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination
>            Reporter: Corentin Chary
>         Attachments: mutation-linkedlist-block.png, profiler-snapshot.nps
>
>
> On a setup with a sustained write load of 70kQPS per node and a RF of 2 it looks like most of the mutation time is spend in OutboundTcpConnection.enqueue() -> backlog.put()
> backlog is an unbounded LinkedBlockingQueue, which means that .put() can only be blocking if a lock is taken. I strongly suspect that this is caused by the use of drainTo() in CoalescingStrategies which is causing contention for the producers.
> On the other hand, not using drainTo() could lead to starvation of the consumers.
> Possible solutions:
> - Allow multiple connections per size and per hosts in OutboundTcpConnectionPool
> - Switch from drainTo to multiple take()
> - Switch to ConcurrentLinkedQueue (which is lockless), also means we need active polling.
> Maybe a good solution would be something hybrid: a bounded LinkedBlockingQueue and an unbounded ConcurrentLinkedQueue. This way you get low latency when you don't have a lot of messages, and throughput when you do.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)