Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9EA1111F9F for ; Tue, 22 Jul 2014 02:23:41 +0000 (UTC) Received: (qmail 35612 invoked by uid 500); 22 Jul 2014 02:23:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 35570 invoked by uid 500); 22 Jul 2014 02:23:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 35559 invoked by uid 99); 22 Jul 2014 02:23:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jul 2014 02:23:39 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dfgriffith@gmail.com designates 74.125.82.174 as permitted sender) Received: from [74.125.82.174] (HELO mail-we0-f174.google.com) (74.125.82.174) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jul 2014 02:23:34 +0000 Received: by mail-we0-f174.google.com with SMTP id x48so8412423wes.19 for ; Mon, 21 Jul 2014 19:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1jsgZfo9zDQNrzhS3+Qgg/4p4wlKMzMl6GSM9HZ8y+4=; b=Z6QYgTslzq0t3i/O3iDGyB96keENo1wckITMQcGo37RjiKvF0KiuEsLipgLZZpeYlz bmGxfUu/4eh+CZD58OiUJgBN3r4tKc+VNI7IQ84oeUmghyOpICwu2LAlOjyVCyk3IuT1 1yPgo7PNr5CyAdPs4cJJtOT5KFiv+xm1eqHk7l4TxexGmOOY6oP6mz1OZ2XNQZbnJ58b 4WApQhkvXjNGDdpZOLuGR70Vtts6EtEWPbzeZrSO53cAjJfJGZVaIxhfqVMXgNDd4Gk9 PqI/xegKk3Sa7JhbqCA7NMzIzoyZXYRCUET2NziF2g13DaI8DOKZy+DCUGKAScqg4hk6 IEAg== MIME-Version: 1.0 X-Received: by 10.180.109.161 with SMTP id ht1mr1708297wib.67.1405995793472; Mon, 21 Jul 2014 19:23:13 -0700 (PDT) Received: by 10.216.105.143 with HTTP; Mon, 21 Jul 2014 19:23:13 -0700 (PDT) In-Reply-To: References: <62C7CCE5A334433FBC9F93F87370648F@JackKrupansky14> <43A7F5BFFCEB4231B2D261DB7F61D648@JackKrupansky14> Date: Mon, 21 Jul 2014 22:23:13 -0400 Message-ID: Subject: Re: horizontal query scaling issues follow on From: Diane Griffith To: user Content-Type: multipart/alternative; boundary=e89a8f23568bd3641404febee734 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f23568bd3641404febee734 Content-Type: text/plain; charset=UTF-8 So I appreciate all the help so far. Upfront, it is possible the schema and data query pattern could be contributing to the problem. The schema was born out of certain design requirements. If it proves to be part of what makes the scalability crumble, then I hope it will help shape the design requirements. Anyway, the premise of the question was my struggle where scalability metrics fell apart going from 2 nodes to 4 nodes for the current schema and query access pattern being modeled: - 1 node was producing acceptable response times seemed to be the consensus - 2 nodes showed marked improvement to the response times for the query scenario being modeled which was welcomed news - 4 nodes showed a decrease in performance and it was not clear why going 2 to 4 nodes triggered the decrease Also what contributed to the question was 2 more items: - cassandra-env.sh - where in the example for HEAP_NEWSIZE states in the comments it assumes a modern 8 core machine for pause times - a wiki article I had found and I am trying to relocate where a person set up very small nodes for developers on that team and talked through all the paramters that had to be changed from the default to get good throughput. It sort of implied the defaults maybe were based on a certain sized vm. That was the main driver for those questions. I agree it does not seem correct to boost the values let alone so high to minimize impact in some respects (i.e. not trigger the reads to time out and start over given the retry policy). So the question really was are the defaults sized with the assumption of a certain minimal vm size (i.e. the comment in cassandra-env.sh) Does that explain where I am coming from better? My question, despite being naive and ignoring other impacts still stands, is there a minimal vm size that is more of the sweet spot for cassandra and the defaults. I get the point that a column family schema as it relates to the desired queries can and do impact that answer. I guess what bothered me was it didn't impact that answer going from 1 node to 2 nodes but started showing up going from 2 nodes to 4 nodes. I'm building whatever facts I can to support the schema and query pattern scales or does not. If it does not, then I am trying to pull information from some metrics outputted by nodetool or log statements on the cassandra log files to support a case to change the design requirements. Thanks, Diane On Mon, Jul 21, 2014 at 8:15 PM, Robert Coli wrote: > On Sun, Jul 20, 2014 at 6:12 PM, Diane Griffith > wrote: > >> I am running tests again across different number of client threads and >> number of nodes but this time I tweaked some of the timeouts configured for >> the nodes in the cluster. I was able to get better performance on the >> nodes at 10 client threads by upping 4 timeout values in cassandra.yaml to >> 240000: >> > > If you have to tune these timeout values, you have probably modeled data > in such a way that each of your requests is "quite large" or "quite slow". > > This is usually, but not always, an indicator that you are Doing It Wrong. > Massively multithreaded things don't generally like their threads to be > long-lived, for what should hopefully be obvious reasons. > > >> I did this because of my interpretation of the cfhistograms output on one >> of the nodes. >> > > Could you be more specific? > > >> So 3 questions that come to mind: >> >> >> 1. Did I interpret the histogram information correctly in cassandra >> 2.0.6 nodetool output? That the 2 column read latency output is the offset >> or left column is the time in milliseconds and the right column is number >> of requests that fell into that bucket range. >> 2. Was it reasonable for me to boost those 4 timeouts and just those? >> >> Not really. In 5 years of operating Cassandra, I've never had a problem > whose solution was to increase these timeouts from their default. > >> >> 1. What are reasonable timeout values for smaller vm sizes (i.e. 8GB >> RAM, 4 CPUs)? >> >> As above, I question the premise of this question. > > =Rob > > --e89a8f23568bd3641404febee734 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
So I appreciate all the help so far. =C2=A0Upfront, it is = possible the schema and data query pattern could be contributing to the pro= blem. =C2=A0The schema was born out of certain design requirements. =C2=A0I= f it proves to be part of what makes the scalability crumble, then I hope i= t will help shape the design requirements.

Anyway, the premise of the question was my struggle where sc= alability metrics fell apart going from 2 nodes to 4 nodes for the current = schema and query access pattern being modeled:
- 1 node was produ= cing acceptable response times seemed to be the consensus=C2=A0
- 2 nodes showed marked improvement to the response times for the quer= y scenario being modeled which was welcomed news
- 4 nodes showed= a decrease in performance and it was not clear why going 2 to 4 nodes trig= gered the decrease

Also what contributed to the question was 2 more items:=
- cassandra-env.sh - where in the example for HEAP_NEWSIZE state= s in the comments it assumes a modern 8 core machine for pause times
- a wiki article I had found and I am trying to relocate where a perso= n set up very small nodes for developers on that team and talked through al= l the paramters that had to be changed from the default to get good through= put. =C2=A0It sort of implied the defaults maybe were based on a certain si= zed vm.

That was the main driver for those questions. I agree i= t does not seem correct to boost the values let alone so high to minimize i= mpact in some respects (i.e. not trigger the reads to time out and start ov= er given the retry policy). =C2=A0

So the question really was are the defaults sized with = the assumption of a certain minimal vm size (i.e. the comment in cassandra-= env.sh)

Does that explain where I am coming from b= etter?

My question, despite being naive and ignoring oth= er impacts still stands, is there a minimal vm size that is more of the swe= et spot for cassandra and the defaults. =C2=A0I get the point that a column= family schema as it relates to the desired queries can and do impact that = answer. =C2=A0I guess what bothered me was it didn't impact that answer= going from 1 node to 2 nodes but started showing up going from 2 nodes to = 4 nodes. =C2=A0

I'm building whatever facts I can to support the sc= hema and query pattern scales or does not. =C2=A0If it does not, then I am = trying to pull information from some metrics outputted by nodetool or log s= tatements on the cassandra log files to support a case to change the design= requirements. =C2=A0

Thanks,
Diane


On Mon, Jul 21, 2014 at 8:15 PM, = Robert Coli <rcoli@eventbrite.com> wrote:
=
On Sun, Jul 20, 2014 at 6:12 PM,= Diane Griffith <dfgriffith@gmail.com> wrote:
I am running tests again ac= ross different number of client threads and number of nodes but this time I= tweaked some of the timeouts configured for the nodes in the cluster. =C2= =A0I was able to get better performance on the nodes at 10 client threads b= y upping 4 timeout values in cassandra.yaml to 240000:
=C2=A0
If you have to tune these timeout = values, you have probably modeled data in such a way that each of your requ= ests is "quite large" or "quite slow".

This is usually, but not always, an indicator that you are Doing It Wrong. = Massively multithreaded things don't generally like their threads to be= long-lived, for what should hopefully be obvious reasons.
=C2=A0
I did this becau= se of my interpretation of the cfhistograms output on one of the nodes. =C2= =A0

Could you be more specific?
=C2=A0
So 3 questions that come to mind:

  1. Did I interpret the histogram information corre= ctly in cassandra 2.0.6 nodetool output? =C2=A0That the 2 column read laten= cy output is the offset or left column is the time in milliseconds and the = right column is number of requests that fell into that bucket range.
  2. Was it reasonable for me to boost those 4 timeouts and just those?=
Not really. In 5 years of ope= rating Cassandra, I've never had a problem whose solution was to increa= se these timeouts from their default.
  1. What are reaso= nable timeout values for smaller vm sizes (i.e. 8GB RAM, 4 CPUs)? =C2=A0
As above, I question the premise of thi= s question.

=3DRob


--e89a8f23568bd3641404febee734--