Return-Path: X-Original-To: apmail-cassandra-dev-archive@www.apache.org Delivered-To: apmail-cassandra-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5E960106DA for ; Thu, 27 Feb 2014 21:07:19 +0000 (UTC) Received: (qmail 28332 invoked by uid 500); 27 Feb 2014 21:07:18 -0000 Delivered-To: apmail-cassandra-dev-archive@cassandra.apache.org Received: (qmail 28282 invoked by uid 500); 27 Feb 2014 21:07:17 -0000 Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list dev@cassandra.apache.org Received: (qmail 28270 invoked by uid 99); 27 Feb 2014 21:07:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 21:07:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of belliottsmith@datastax.com designates 209.85.223.169 as permitted sender) Received: from [209.85.223.169] (HELO mail-ie0-f169.google.com) (209.85.223.169) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 21:07:11 +0000 Received: by mail-ie0-f169.google.com with SMTP id at1so2638390iec.28 for ; Thu, 27 Feb 2014 13:06:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Va1AxFax3Xuvszp4GhylrnSRqVStUYO9ugoOzztcGxs=; b=hPh6yCq7WJ6XuAQYeW+S3uw0UEyT1hurDB8AlVKNf/tCJJn6cUOmHvw4GMTMYsbaMH mfuXvZfdIeX+bEHoTkeWPeHtrfUf43m6DIfYYwtBZ98RKPIJ9XHuetSu1xNiGTej8wWc 0TM/KnrwN/51uWQx7z23ftleDCrkxYGUUQsPulwA1GNqdu8lE3xCaU7GVahwtApLohhz 3Y0iPKfywySz2jJ93qt8HOm00Z05DMSsyKC0wM5aIIiH/HjM7pxJxLkEqVzT7qMzbmus EcJVd+HqR18O/IgFpk0Z0+OqALEmF11ire1iRbVf8lVM5XzZLZa2R+/DlTGivs8f15tu dyZw== X-Gm-Message-State: ALoCoQkYcmnA7FlMAkfz5CsSwg5b23hkeWhVFQbh49xh/x0+CdP+eINtvjzFXBN+tXniRwXHsLLF MIME-Version: 1.0 X-Received: by 10.50.62.178 with SMTP id z18mr5925igr.49.1393535210018; Thu, 27 Feb 2014 13:06:50 -0800 (PST) Received: by 10.43.98.202 with HTTP; Thu, 27 Feb 2014 13:06:49 -0800 (PST) Received: by 10.43.98.202 with HTTP; Thu, 27 Feb 2014 13:06:49 -0800 (PST) In-Reply-To: References: Date: Thu, 27 Feb 2014 21:06:49 +0000 Message-ID: Subject: Re: [jira] [Commented] (CASSANDRA-6746) Reads have a slow ramp up in speed From: Benedict Elliott Smith To: dev@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7bd760c22cf39204f369b3e9 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd760c22cf39204f369b3e9 Content-Type: text/plain; charset=ISO-8859-1 I've been investigating this, but a bit slowly as was looking in the wrong place. I haven't yet confirmed, but I have a strong suspicion of the problem. Could you confirm the total physical memory on the nodes? If 8gb or less, try applying the not yet committed patches in CASSANDRA-6692 (atomic b tree improvements), and setting the memtable_cleanup_threshold to 0.2, if you are looking at this today. I suspect it will fix it, although if so it doesn't quite adequately explain the behaviour exactly. If you let me know the cluster you're using I can test the same environment to make sure I'm testing like for like as well. On 27 Feb 2014 20:58, "Ryan McGuire (JIRA)" wrote: > > [ > https://issues.apache.org/jira/browse/CASSANDRA-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915038#comment-13915038] > > Ryan McGuire commented on CASSANDRA-6746: > ----------------------------------------- > > I tried re-running this with a 'nodetool flush' on each node after it's > done with the write. It looked the same as above. I'm running a test with a > 5 minute wait between the write and read to see if that causes a change. > > > Reads have a slow ramp up in speed > > ---------------------------------- > > > > Key: CASSANDRA-6746 > > URL: > https://issues.apache.org/jira/browse/CASSANDRA-6746 > > Project: Cassandra > > Issue Type: Bug > > Components: Core > > Reporter: Ryan McGuire > > Assignee: Benedict > > Labels: performance > > Fix For: 2.1 beta2 > > > > Attachments: 2.1_vs_2.0_read.png > > > > > > On a physical four node cluister I am doing a big write and then a big > read. The read takes a long time to ramp up to respectable speeds. > > !2.1_vs_2.0_read.png! > > [See data here| > http://ryanmcguire.info/ds/graph/graph.html?stats=stats.2.1_vs_2.0_vs_1.2.retry1.json&metric=interval_op_rate&operation=stress-read&smoothing=1 > ] > > > > -- > This message was sent by Atlassian JIRA > (v6.1.5#6160) > --047d7bd760c22cf39204f369b3e9--