Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9C80A9D39 for ; Mon, 5 Mar 2012 11:40:52 +0000 (UTC) Received: (qmail 3010 invoked by uid 500); 5 Mar 2012 11:40:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 2991 invoked by uid 500); 5 Mar 2012 11:40:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 2983 invoked by uid 99); 5 Mar 2012 11:40:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 11:40:50 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Dushyant.Tiwari@morganstanley.com designates 205.228.53.68 as permitted sender) Received: from [205.228.53.68] (HELO hqmtaint01.ms.com) (205.228.53.68) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 11:40:44 +0000 Received: from hqmtaint01.ms.com (localhost.ms.com [127.0.0.1]) by hqmtaint01.ms.com (output Postfix) with ESMTP id D70C85045FC for ; Mon, 5 Mar 2012 06:40:22 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=morganstanley.com; s=p20110615; t=1330947622; x=1332157222; bh=2plGjOMjaq/TTUkZKD8z7La7VPPjqYQOViNeDpQCAm0=; h=Content-Transfer-Encoding:From:To:Date:Subject:Message-ID: References:In-Reply-To:Content-Type:MIME-Version; b=lsKTwTALDlShmmJ1xaBXRZ6QaFW1H35GRwE+8PoKERx34hb3qmkjWnNCgkuNIpZ6A Ca0B2HrpUxVwLzpoRVIIQQXJj50QyoWMwJVKl3obNFe1afZ1UmGYoV9HpLAZ2jokSC BkzVXuoxA2EXSZ39BrKIfYSO1uiKS7Nb6nJ44FpU= Received: from ny021bas01.ms.com (ny021bas01.ms.com [10.162.156.33]) by hqmtaint01.ms.com (internal Postfix) with ESMTP id D48315045E3 for ; Mon, 5 Mar 2012 06:40:22 -0500 (EST) Received: from ny021bas01.ms.com (localhost [127.0.0.1]) by ny021bas01.ms.com (msa-out Postfix) with ESMTP id BDE52FB4013 for ; Mon, 5 Mar 2012 06:40:22 -0500 (EST) Received: from HNWEXGOB02.msad.ms.com (hn212c1n1 [10.184.121.167]) by ny021bas01.ms.com (mta-in Postfix) with ESMTP id BAE0BDE003A for ; Mon, 5 Mar 2012 06:40:22 -0500 (EST) Received: from tiwexcat02.msad.ms.com (161.144.143.23) by HNWEXGOB02.msad.ms.com (10.184.121.167) with Microsoft SMTP Server (TLS) id 8.3.213.0; Mon, 5 Mar 2012 06:40:22 -0500 Received: from TIWEXMBX13.msad.ms.com ([161.144.143.13]) by tiwexcat02.msad.ms.com ([161.144.143.23]) with mapi; Mon, 5 Mar 2012 17:10:19 +0530 Content-Transfer-Encoding: 7bit From: "Tiwari, Dushyant" Content-Class: urn:content-classes:message To: Importance: normal Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4913 Date: Mon, 5 Mar 2012 17:10:18 +0530 Subject: RE: Mutation Dropped Messages Thread-Topic: Mutation Dropped Messages thread-index: Acz6vQmhlWD7cDeFSh+4qgr4HmxNvgAB1yHw Message-ID: References: <802222A2-ECF8-42FA-8767-2B9BBAFEBEAD@thelastpickle.com> In-Reply-To: <802222A2-ECF8-42FA-8767-2B9BBAFEBEAD@thelastpickle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE0194891905F7TIWEXMBX13m_" MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: be3bdf5c-71de-49fb-a1b7-ad6a6a1df5e2 X-Virus-Checked: Checked by ClamAV on apache.org --_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE0194891905F7TIWEXMBX13m_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks a lot for the concurrent_writes hint that really improves the = throughput. Do you mean dropped messages and no timedoutexception will = mean the data is written somewhere in the cluster and by taking = corrective measures desired CL can be achieved? From: aaron morton [mailto:aaron@thelastpickle.com] Sent: Monday, March 05, 2012 4:15 PM To: user@cassandra.apache.org Subject: Re: Mutation Dropped Messages 1. Which parameters to tune in the config files? - Especially = looking for heavy writes The node is overloaded. It may be because there are no enough nodes, or = the node is under temporary stress such as GC or repair. If you have spare IO / CPU capacity you could increase the = current_writes to increase throughput on the write stage. You then need = to ensure the commit log and, to a lesser degree, the data volumes can = keep up. 2. What is the difference between TimedOutException and silently = dropping mutation messages while operating on a CL of QUORUM. TimedOutExceptions means CL nodes did not respond to the coordinator = before rpc_timeout. Dropping messages happens when a message is removed = from the queue in the a thread pool after rpc_timeout has occurred. it = is a feature of the architecture, and correct behaviour under stress. Inconsistencies created by dropped messages are repaired via reads as = high CL, HH (in 1.+), Read Repair or Anti Entropy. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/03/2012, at 11:32 PM, Tiwari, Dushyant wrote: Hi All, While benchmarking Cassandra I found "Mutation Dropped" messages in the = logs. Now I know this is a good old question. It will be really great = if someone can provide a check list to recover when such a thing = happens. I am looking for answers of the following questions - 1. Which parameters to tune in the config files? - Especially = looking for heavy writes 2. What is the difference between TimedOutException and silently = dropping mutation messages while operating on a CL of QUORUM. Regards, Dushyant ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. If you have received = this communication in error, please destroy all electronic and paper = copies and notify the sender immediately. Mistransmission is not = intended to waive confidentiality or privilege. Morgan Stanley reserves = the right, to the extent permitted under applicable law, to monitor = electronic communications. This message is subject to terms available at = the following link: http://www.morganstanley.com/disclaimers. If you = cannot access these links, please notify us by reply message and we will = send the contents to you. By messaging with Morgan Stanley you consent = to the foregoing. -------------------------------------------------------------------------= - NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. If you have received = this communication in error, please destroy all electronic and paper = copies and notify the sender immediately. Mistransmission is not = intended to waive confidentiality or privilege. Morgan Stanley reserves = the right, to the extent permitted under applicable law, to monitor = electronic communications. This message is subject to terms available at = the following link: http://www.morganstanley.com/disclaimers. If you = cannot access these links, please notify us by reply message and we will = send the contents to you. By messaging with Morgan Stanley you consent = to the foregoing. --_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE0194891905F7TIWEXMBX13m_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks a lot for the concurrent_writes hint that really improves the = throughput. Do you mean dropped messages and no timedoutexception will = mean the data is written somewhere in the cluster and by taking = corrective measures desired CL can be achieved?

 

 

 

From:= = aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, = March 05, 2012 4:15 PM
To: = user@cassandra.apache.org
Subject: Re: Mutation Dropped = Messages

 

1.      = ; Which parameters to tune in the config files? – Especially = looking for heavy = writes

The node is overloaded. It may be because there are no = enough nodes, or the node is under temporary stress such as GC or = repair. 

If you have spare = IO / CPU capacity you could increase the current_writes to increase = throughput on the write stage. You then need to ensure the commit log = and, to a lesser degree, the data volumes can keep = up. 

 

2.      = ; What is the difference between TimedOutException and silently dropping = mutation messages while operating on a CL of = QUORUM.

TimedOutExceptions means CL nodes did not respond to = the coordinator before rpc_timeout. Dropping messages happens when a = message is removed from the queue in the a thread pool after rpc_timeout = has occurred. it is a feature of the architecture, and correct behaviour = under stress. 

Inconsistencies created by dropped messages are = repaired via reads as high CL, HH (in 1.+), Read Repair or Anti = Entropy.

 

Cheers

 

-----------------

Aaron Morton

Freelance Developer

@aaronmorton

 

On = 5/03/2012, at 11:32 PM, Tiwari, Dushyant wrote:



Hi All,

 

While benchmarking Cassandra I found “Mutation Dropped” = messages in the logs.  Now I know this is a good old question. It = will be really great if someone can provide a check list to recover when = such a thing happens. I am looking for answers of the following = questions  -

 

1.      = ; Which parameters to tune in the config files? – Especially = looking for heavy writes

2.      = ; What is the difference between TimedOutException and silently dropping = mutation messages while operating on a CL of = QUORUM.

 

 

Regards,

Dushyant

NOT= ICE: Mor= gan Stanley is not acting as a municipal advisor and the opinions or = views contained herein are not intended to be, and do not constitute, = advice within the meaning of Section 975 of the Dodd-Frank Wall Street = Reform and Consumer Protection Act. If = you have received this communication in error, please destroy all = electronic and paper copies and notify the sender immediately. = Mistransmission is not intended to waive confidentiality or privilege. = Morgan Stanley reserves the right, to the extent permitted under = applicable law, to monitor electronic communications. This message is = subject to terms available at the following link: http://www.morganstanley.com/disclai= mers. = If you cannot access these links, please notify us by reply message and = we will send the contents to you. By messaging with Morgan Stanley you = consent to the foregoing.

 

<= /FONT>
NOTICE: Morgan Stanley is not acting as a municipal advisor and the = opinions or views contained herein are not intended to be, and do not = constitute, advice within the meaning of Section 975 of the Dodd-Frank = Wall Street Reform and Consumer Protection Act. = If you = have received this communication in error, please destroy all electronic = and paper copies and notify the sender immediately. Mistransmission is = not intended to waive confidentiality or privilege. Morgan Stanley = reserves the right, to the extent permitted under applicable law, to = monitor electronic communications. This message is subject to terms = available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please = notify us by reply message and we will send the contents to you. By = messaging with Morgan Stanley you consent to the = foregoing.
--_000_A7C1BA1FB68BAA489E8BAF7E7F5889EE0194891905F7TIWEXMBX13m_--