Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7BE4D10D9E for ; Wed, 12 Mar 2014 23:38:32 +0000 (UTC) Received: (qmail 50446 invoked by uid 500); 12 Mar 2014 23:38:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50388 invoked by uid 500); 12 Mar 2014 23:38:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50380 invoked by uid 99); 12 Mar 2014 23:38:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Mar 2014 23:38:29 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dmcnelis@gmail.com designates 209.85.192.44 as permitted sender) Received: from [209.85.192.44] (HELO mail-qg0-f44.google.com) (209.85.192.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Mar 2014 23:38:24 +0000 Received: by mail-qg0-f44.google.com with SMTP id a108so730856qge.3 for ; Wed, 12 Mar 2014 16:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=6I5KN0Mc9/EFCRZs+90pKKLlihXrtGKG8nrGgDo9l7U=; b=JjK1aJy/Fe56zJuVM0UQc+oqNOvvQmbIPBTIBDloBcBEu0gHCQxyJLlGO+M7IjrumS ZSmPS7hDrgsl9aCoNa6Apjw9jje2KMkh7v5JN41vMjQFn13L8vCUrVRLE5lDIgE7XL0z N3mKR3vbzsyvQtn48C9rhvZffSIm5ceS3MO+2ZDrR+6MM6A4Rcy8odrxQt1j8bLAQJ4A pa6w1B6gzKB5VomdvtozSNw20u/Xfss+ta1Wcn0wtqSDDpA09FJ7d2kg2I3itRo8iQcW s3Dgxg+jGrTA7RJPnSvue2fk7gbi75R7gJSnYwZnDGZJcsZh1tTtROddzPnyGsDlC1iD qP2Q== MIME-Version: 1.0 X-Received: by 10.224.66.74 with SMTP id m10mr421374qai.14.1394667483472; Wed, 12 Mar 2014 16:38:03 -0700 (PDT) Received: by 10.229.57.200 with HTTP; Wed, 12 Mar 2014 16:38:03 -0700 (PDT) In-Reply-To: <7AE403FB-D457-483E-9305-B475413AA4F5@gmail.com> References: <1394656015.88878.YahooMailNeo@web164502.mail.gq1.yahoo.com> <7AE403FB-D457-483E-9305-B475413AA4F5@gmail.com> Date: Wed, 12 Mar 2014 19:38:03 -0400 Message-ID: Subject: Re: From: David McNelis To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=001a11c2c338eec23304f471538f X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2c338eec23304f471538f Content-Type: text/plain; charset=ISO-8859-1 Not knowing anything about your data structure (to expand on what Edward said), you could be running into something where you've got some hot keys that are getting the majority of writes during those heavily loads.... more specifically I might look for a single key that you're writing, since you're RF=3 and you have 3 nodes specifically that are causing problems. On Wed, Mar 12, 2014 at 7:28 PM, Russ Bradberry wrote: > I wouldn't go above 8G unless you have a very powerful machine that can > keep the GC pauses low. > > Sent from my iPhone > > On Mar 12, 2014, at 7:11 PM, Edward Capriolo > wrote: > > That is too much ram for cassandra make that 6g to 10g. > > The uneven perf could be because your requests do not shard evenly. > > On Wednesday, March 12, 2014, Batranut Bogdan wrote: > > Hello all, > > > > The environment: > > > > I have a 6 node Cassandra cluster. On each node I have: > > - 32 G RAM > > - 24 G RAM for cassa > > - ~150 - 200 MB/s disk speed > > - tomcat 6 with axis2 webservice that uses the datastax java driver to > make > > asynch reads / writes > > - replication factor for the keyspace is 3 > > > > All nodes in the same data center > > The clients that read / write are in the same datacenter so network is > > Gigabit. > > > > Writes are performed via exposed methods from Axis2 WS . The Cassandra > Java > > driver uses the round robin load balancing policy so all the nodes in the > > cluster should be hit with write requests under heavy write or read load > > from multiple clients. > > > > I am monitoring all nodes with JConsole from another box. > > > > The problem: > > > > When wrinting to a particular column family, only 3 nodes have high CPU > load > > ~ 80 - 99 %. The remaining 3 are at ~2 - 10 % CPU. During writes, reads > > timeout. > > > > I need more speed for both writes of reads. Due to the fact that 3 nodes > > barely have CPU activity leads me to think that the whole potential for > C* > > is not touched. > > > > I am running out of ideas... > > > > If further details about the environment I can provide them. > > > > > > Thank you very much. > > -- > Sorry this was sent from mobile. Will do less grammar and spell check than > usual. > > --001a11c2c338eec23304f471538f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Not knowing anything about your data structure (to expand = on what Edward said), you could be running into something where you've = got some hot keys that are getting the majority of writes during those heav= ily loads.... more specifically I might look for a single key that you'= re writing, since you're RF=3D3 and you have 3 nodes specifically that = are causing problems.



On Wed, Mar 12, 2014 at 7:28 PM, Russ Bradberry <rbradberry@gmail.co= m> wrote:
I wouldn't go abo= ve 8G unless you have a very powerful machine that can keep the GC pauses l= ow.

Sent from my iPhone

On Mar 12, 201= 4, at 7:11 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:

That is too much ram for cassandra make that 6g to 10g.

The un= even perf could be because your requests do not shard evenly.

On Wed= nesday, March 12, 2014, Batranut Bogdan <batranub@yahoo.com> wrote:
> Hello all,
>
> The environment:
>
> I have a 6= node Cassandra cluster. On each node I have:
> - 32 G RAM
> - = 24 G RAM for cassa
> - ~150 - 200 MB/s disk speed
> - tomcat 6 = with axis2 webservice that uses the datastax java driver to make
> asynch reads / writes=A0
> - replication factor for the keyspace= is 3
>
> All nodes in the same data center=A0
> The clie= nts that read / write are in the same datacenter so network is
> Giga= bit.
>
> Writes are performed via exposed methods from Axis2 WS . The C= assandra Java
> driver uses the round robin load balancing policy so = all the nodes in the
> cluster should be hit with write requests unde= r heavy write or read load
> from multiple clients.
>
> I am monitoring all nodes with = JConsole from another box.
>
> The problem:
>
> Whe= n wrinting to a particular column family, only 3 nodes have high CPU load > ~ 80 - 99 %. The remaining 3 are at ~2 - 10 % CPU. During writes, read= s
> timeout.=A0
>
> I need more speed for both writes of = reads. Due to the fact that 3 nodes
> barely have CPU activity leads = me to think that the whole potential for C*
> is not touched.
>
> I am running out of ideas...
>> If further details about the environment I can provide them.
>=
>
> Thank you very much.

--
Sorry this was sent fro= m mobile. Will do less grammar and spell check than usual.

--001a11c2c338eec23304f471538f--