Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3EFCE95AD for ; Mon, 5 Mar 2012 14:58:56 +0000 (UTC) Received: (qmail 7142 invoked by uid 500); 5 Mar 2012 14:58:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 7114 invoked by uid 500); 5 Mar 2012 14:58:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 7106 invoked by uid 99); 5 Mar 2012 14:58:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 14:58:53 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Dushyant.Tiwari@morganstanley.com designates 205.228.53.73 as permitted sender) Received: from [205.228.53.73] (HELO hqmtaint03.ms.com) (205.228.53.73) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 14:58:47 +0000 Received: from hqmtaint03.ms.com (localhost.ms.com [127.0.0.1]) by hqmtaint03.ms.com (output Postfix) with ESMTP id 5D732B6C915 for ; Mon, 5 Mar 2012 09:58:26 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=morganstanley.com; s=p20110615; t=1330959506; x=1332169106; bh=79Tywfl3hjO+P+JwcMrXakGPm53xjJXRMhxGnlrHh7I=; h=Content-Transfer-Encoding:From:To:Date:Subject:Message-ID: References:In-Reply-To:Content-Type:MIME-Version; b=syxIngtdZ3FlxmuI4VlzmjCl0k2mMvVnjLjpP8S1BJxCuuseDELTdqTbzrrCNzONe OZ4vq1tdUTcNCLfH9j/U2qm2C3AuU6t/dZ3BHkB6YE8kluP+ADTB8EN2UuSgAxocSU Ir7/6WnMHlOhI91wWZCe+nJMfcZYGjmzyKFcwXIM= Received: from ny021bas02.ms.com (ny021bas02.ms.com [10.162.156.34]) by hqmtaint03.ms.com (internal Postfix) with ESMTP id 5B212A30D0E for ; Mon, 5 Mar 2012 09:58:26 -0500 (EST) Received: from ny021bas02.ms.com (localhost [127.0.0.1]) by ny021bas02.ms.com (msa-out Postfix) with ESMTP id 4294254080 for ; Mon, 5 Mar 2012 09:58:26 -0500 (EST) Received: from NPWEXGOB01.msad.ms.com (np210c1n1 [10.184.90.162]) by ny021bas02.ms.com (mta-in Postfix) with ESMTP id 3FEE1748034 for ; Mon, 5 Mar 2012 09:58:26 -0500 (EST) Received: from tiwexcat02.msad.ms.com (161.144.143.23) by NPWEXGOB01.msad.ms.com (10.184.90.162) with Microsoft SMTP Server (TLS) id 8.3.213.0; Mon, 5 Mar 2012 09:58:25 -0500 Received: from TIWEXMBX13.msad.ms.com ([161.144.143.13]) by tiwexcat02.msad.ms.com ([161.144.143.23]) with mapi; Mon, 5 Mar 2012 20:28:22 +0530 Content-Transfer-Encoding: 7bit From: "Tiwari, Dushyant" To: Content-Class: urn:content-classes:message Importance: normal Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4913 Date: Mon, 5 Mar 2012 20:28:21 +0530 Subject: RE: Mutation Dropped Messages Thread-Topic: Mutation Dropped Messages thread-index: Acz6vQmhlWD7cDeFSh+4qgr4HmxNvgAIoDeQ Message-ID: References: <802222A2-ECF8-42FA-8767-2B9BBAFEBEAD@thelastpickle.com> In-Reply-To: <802222A2-ECF8-42FA-8767-2B9BBAFEBEAD@thelastpickle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE01948919063DTIWEXMBX13m_" MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: be3bdf5c-71de-49fb-a1b7-ad6a6a1df5e2 X-Virus-Checked: Checked by ClamAV on apache.org --_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE01948919063DTIWEXMBX13m_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hey Aaron, I increased the size of the cluster also the concurrent_writes = parameter. Still there is a node which keeps on dropping the mutation = messages. The other nodes are not dropping mutation messages. I am using = Hector API and had done nothing for load balancing so far. Just provided = the host:port of the nodes in the Cassandrahostconfig. Is this due to = some improper load balancing? Also the physical host where the node is = hosted is relatively heavier than other nodes' host. What can I do to = improve? PS: The node is seed of the cluster. Thanks, Dushyant From: aaron morton [mailto:aaron@thelastpickle.com] Sent: Monday, March 05, 2012 4:15 PM To: user@cassandra.apache.org Subject: Re: Mutation Dropped Messages 1. Which parameters to tune in the config files? - Especially = looking for heavy writes The node is overloaded. It may be because there are no enough nodes, or = the node is under temporary stress such as GC or repair. If you have spare IO / CPU capacity you could increase the = current_writes to increase throughput on the write stage. You then need = to ensure the commit log and, to a lesser degree, the data volumes can = keep up. 2. What is the difference between TimedOutException and silently = dropping mutation messages while operating on a CL of QUORUM. TimedOutExceptions means CL nodes did not respond to the coordinator = before rpc_timeout. Dropping messages happens when a message is removed = from the queue in the a thread pool after rpc_timeout has occurred. it = is a feature of the architecture, and correct behaviour under stress. Inconsistencies created by dropped messages are repaired via reads as = high CL, HH (in 1.+), Read Repair or Anti Entropy. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/03/2012, at 11:32 PM, Tiwari, Dushyant wrote: Hi All, While benchmarking Cassandra I found "Mutation Dropped" messages in the = logs. Now I know this is a good old question. It will be really great = if someone can provide a check list to recover when such a thing = happens. I am looking for answers of the following questions - 1. Which parameters to tune in the config files? - Especially = looking for heavy writes 2. What is the difference between TimedOutException and silently = dropping mutation messages while operating on a CL of QUORUM. Regards, Dushyant ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. If you have received = this communication in error, please destroy all electronic and paper = copies and notify the sender immediately. Mistransmission is not = intended to waive confidentiality or privilege. Morgan Stanley reserves = the right, to the extent permitted under applicable law, to monitor = electronic communications. This message is subject to terms available at = the following link: http://www.morganstanley.com/disclaimers. If you = cannot access these links, please notify us by reply message and we will = send the contents to you. By messaging with Morgan Stanley you consent = to the foregoing. -------------------------------------------------------------------------= - NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. If you have received = this communication in error, please destroy all electronic and paper = copies and notify the sender immediately. Mistransmission is not = intended to waive confidentiality or privilege. Morgan Stanley reserves = the right, to the extent permitted under applicable law, to monitor = electronic communications. This message is subject to terms available at = the following link: http://www.morganstanley.com/disclaimers. If you = cannot access these links, please notify us by reply message and we will = send the contents to you. By messaging with Morgan Stanley you consent = to the foregoing. --_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE01948919063DTIWEXMBX13m_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hey Aaron,

 

I increased the size of the cluster also the concurrent_writes = parameter. Still there is a node which keeps on dropping the mutation = messages. The other nodes are not dropping mutation messages. I am using = Hector API and had done nothing for load balancing so far. Just provided = the host:port of the nodes in the Cassandrahostconfig. Is this due to = some improper load balancing? Also the physical host where the node is = hosted is relatively heavier than other nodes’ host. What can I do = to improve?

PS: The node is seed of the cluster.

 

Thanks,

Dushyant

 

From:= = aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, = March 05, 2012 4:15 PM
To: = user@cassandra.apache.org
Subject: Re: Mutation Dropped = Messages

 

1.      = ; Which parameters to tune in the config files? – Especially = looking for heavy = writes

The node is overloaded. It may be because there are no = enough nodes, or the node is under temporary stress such as GC or = repair. 

If you have spare = IO / CPU capacity you could increase the current_writes to increase = throughput on the write stage. You then need to ensure the commit log = and, to a lesser degree, the data volumes can keep = up. 

 

2.      = ; What is the difference between TimedOutException and silently dropping = mutation messages while operating on a CL of = QUORUM.

TimedOutExceptions means CL nodes did not respond to = the coordinator before rpc_timeout. Dropping messages happens when a = message is removed from the queue in the a thread pool after rpc_timeout = has occurred. it is a feature of the architecture, and correct behaviour = under stress. 

Inconsistencies created by dropped messages are = repaired via reads as high CL, HH (in 1.+), Read Repair or Anti = Entropy.

 

Cheers

 

-----------------

Aaron Morton

Freelance Developer

@aaronmorton

 

On = 5/03/2012, at 11:32 PM, Tiwari, Dushyant wrote:



Hi All,

 

While benchmarking Cassandra I found “Mutation Dropped” = messages in the logs.  Now I know this is a good old question. It = will be really great if someone can provide a check list to recover when = such a thing happens. I am looking for answers of the following = questions  -

 

1.      = ; Which parameters to tune in the config files? – Especially = looking for heavy writes

2.      = ; What is the difference between TimedOutException and silently dropping = mutation messages while operating on a CL of = QUORUM.

 

 

Regards,

Dushyant

NOT= ICE: Mor= gan Stanley is not acting as a municipal advisor and the opinions or = views contained herein are not intended to be, and do not constitute, = advice within the meaning of Section 975 of the Dodd-Frank Wall Street = Reform and Consumer Protection Act. If = you have received this communication in error, please destroy all = electronic and paper copies and notify the sender immediately. = Mistransmission is not intended to waive confidentiality or privilege. = Morgan Stanley reserves the right, to the extent permitted under = applicable law, to monitor electronic communications. This message is = subject to terms available at the following link: http://www.morganstanley.com/disclai= mers. = If you cannot access these links, please notify us by reply message and = we will send the contents to you. By messaging with Morgan Stanley you = consent to the foregoing.

 

<= /FONT>
NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. = If you = have received this communication in error, please destroy all electronic = and paper copies and notify the sender immediately. Mistransmission is = not intended to waive confidentiality or privilege. Morgan Stanley = reserves the right, to the extent permitted under applicable law, to = monitor electronic communications. This message is subject to terms = available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please = notify us by reply message and we will send the contents to you. By = messaging with Morgan Stanley you consent to the = foregoing.
--_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE01948919063DTIWEXMBX13m_--