Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 42E12D653 for ; Fri, 30 Nov 2012 16:12:35 +0000 (UTC) Received: (qmail 60946 invoked by uid 500); 30 Nov 2012 16:12:32 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 60810 invoked by uid 500); 30 Nov 2012 16:12:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 60689 invoked by uid 99); 30 Nov 2012 16:12:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2012 16:12:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yiming.sun@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2012 16:12:24 +0000 Received: by mail-we0-f172.google.com with SMTP id r3so226626wey.31 for ; Fri, 30 Nov 2012 08:12:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=RRAKQM/TmPrpyBALH+UuBuHXF367Jn8034tRW7bhkwU=; b=TEcRjdpei8nsCP1ljlDYv3EK3GeG32TZ9J/v8areB1N4TZFfPWKccFjAhdm/mdp6Tl /00Yb4jPx2a12xxFbnY2DEx60MkItzLa7fBjo+cQuOLGdmQvYt+nxbLAL7Ok17e5fdXw OMXnNAyQv8PvB7gTcw3h6zqWYqDUbVewvce6uSAaAuO0rn0VqsAEcLAcz0Q0TURwHz1P H5I0KLp7tRfe36UVU6QkNHq5DSWVpxH/F2WG1xi9TaE7BnhDpcDzNmd0m0Ekaj2bjFsu WeKZtfzuVIjr/m0Ub0Qihj+vW9cyEtA9VuePKnm//nluoU4sSVFIKQTMNFDX2Dis5u54 KZAw== Received: by 10.216.202.152 with SMTP id d24mr718289weo.117.1354291923795; Fri, 30 Nov 2012 08:12:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.216.143.161 with HTTP; Fri, 30 Nov 2012 08:11:42 -0800 (PST) In-Reply-To: References: From: Yiming Sun Date: Fri, 30 Nov 2012 11:11:42 -0500 Message-ID: Subject: Re: strange row cache behavior To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636c5bb5b0a3ef204cfb8a960 X-Virus-Checked: Checked by ClamAV on apache.org --001636c5bb5b0a3ef204cfb8a960 Content-Type: text/plain; charset=ISO-8859-1 Does anyone have any comments/suggestions for me regarding this? Thanks I am trying to understand some strange behavior of cassandra row cache. We > have a 6-node Cassandra cluster in a single data center on 2 racks, and the > neighboring nodes on the ring are from alternative racks. Each node has > 1GB row cache, with key cache disabled. The cluster uses > PropertyFileSnitch, and the ColumnFamily I fetch from uses > NetworkTopologyStrategy, with replication factor of 2. My client code uses > Hector to fetch a fixed set of rows from cassandra > > What I don't quite understand is even after I ran the client code several > times, there are always some nodes with 0 row cache hits, despite that the > row cache from all nodes are filled and all nodes receive requests. > > Which nodes have 0 hits seem to be strongly related to the following: > > - the set of row keys to fetch > - the order of the set of row keys to fetch > - the list of hosts passed to Hector's CassandraHostConfigurator > - the order of the list of hosts passed to Hector > > Can someone shed some lights on how exactly the row cache works and > hopefully also explain the behavior I have been seeing? I thought if the > fixed set of the rows keys are the only thing I am fetching (each row > should be on the order of 10's of MBs, no more than 100MB), and each node > gets requests, and its row cache is filled, there's gotta be some hits. > Apparent this is not the case. Thanks. > > cluster information: > > Address DC Rack Status State Load > Effective-Ownership Token > > 141784319550391026443072753096570088105 > x.x.x.1 DC1 r1 Up Normal 587.46 GB > 33.33% 0 > x.x.x.2 DC1 r2 Up Normal 591.21 GB > 33.33% 28356863910078205288614550619314017621 > x.x.x.3 DC1 r1 Up Normal 594.97 GB > 33.33% 56713727820156410577229101238628035242 > x.x.x.4 DC1 r2 Up Normal 587.15 GB > 33.33% 85070591730234615865843651857942052863 > x.x.x.5 DC1 r1 Up Normal 590.26 GB > 33.33% 113427455640312821154458202477256070484 > x.x.x.6 DC1 r2 Up Normal 583.21 GB > 33.33% 141784319550391026443072753096570088105 > > > [user@node]$ ./checkinfo.sh > *************** x.x.x.4 > Token : 85070591730234615865843651857942052863 > Gossip active : true > Thrift active : true > Load : 587.15 GB > Generation No : 1354074048 > Uptime (seconds) : 36957 > Heap Memory (MB) : 2027.29 / 3948.00 > Data Center : DC1 > Rack : r2 > Exceptions : 0 > > Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, > NaN recent hit rate, 14400 save period in seconds > Row Cache : size 1072651974 (bytes), capacity 1073741824 (bytes), 0 > hits, 2576 requests, NaN recent hit rate, 0 save period in seconds > > *************** x.x.x.6 > Token : 141784319550391026443072753096570088105 > Gossip active : true > Thrift active : true > Load : 583.21 GB > Generation No : 1354074461 > Uptime (seconds) : 36535 > Heap Memory (MB) : 828.71 / 3948.00 > Data Center : DC1 > Rack : r2 > Exceptions : 0 > > Key Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, > NaN recent hit rate, 14400 save period in seconds > Row Cache : size 1072602906 (bytes), capacity 1073741824 (bytes), 0 > hits, 3194 requests, NaN recent hit rate, 0 save period in seconds > --001636c5bb5b0a3ef204cfb8a960 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Does anyone have any comments/suggestions for me regarding this? =A0Thanks<= div class=3D"gmail_extra">

I am trying to understand some strange behavior of cassandra row cache= . =A0We have a 6-node Cassandra cluster in a single data center on 2 racks,= and the neighboring nodes on the ring are from alternative racks. =A0Each = node has 1GB row cache, with key cache disabled. =A0 The cluster uses Prope= rtyFileSnitch, and the ColumnFamily I fetch from uses NetworkTopologyStrate= gy, with replication factor of 2. =A0My=A0client code uses Hector to fetch = a fixed set of rows from cassandra

What I don't quite understand is even after I ran t= he client code several times, there are always some nodes with 0 row cache = hits, despite that the row cache from all nodes are filled and all nodes re= ceive requests.

Which nodes have 0 hits seem to be strongly related to = the following:

=A0- the set of row keys to fetch
=A0- the order of the set of row keys to fetch
=A0- the = list of hosts passed to Hector's CassandraHostConfigurator
=A0- the order of the list of hosts passed to Hector

Can someone shed some lights on how exactly the row cache works and= hopefully also explain the behavior I have been seeing? =A0I thought if th= e fixed set of the rows keys are the only thing I am fetching (each row sho= uld be on the order of 10's of MBs, no more than 100MB), and each node = gets requests, and its row cache is filled, there's gotta be some hits.= =A0Apparent this is not the case. =A0 Thanks.

cluster information:

Address= =A0=A0=A0=A0=A0=A0=A0=A0 DC=A0=A0=A0=A0=A0=A0=A0=A0=A0 Rack=A0=A0=A0=A0=A0= =A0=A0 Status State=A0=A0 Load=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Effective-O= wnership Token=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 14178431= 9550391026443072753096570088105=A0=A0=A0=A0=A0
x.x.x.1 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r1=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 587.46 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0
x.x.x.2 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r2=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 591.21 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 28356863910078205288614550619314017621= =A0=A0=A0=A0=A0=A0
x.x.x.3 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r1=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 594.97 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 56713727820156410577229101238628035242= =A0=A0=A0=A0=A0=A0
x.x.x.4 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r2=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 587.15 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 85070591730234615865843651857942052863= =A0=A0=A0=A0=A0=A0
x.x.x.5 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r1=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 590.26 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 113427455640312821154458202477256070484= =A0=A0=A0=A0=A0
x.x.x.6 =A0 =A0DC1=A0=A0=A0=A0=A0=A0=A0=A0 r2=A0=A0=A0=A0=A0=A0=A0=A0=A0 U= p=A0=A0=A0=A0 Normal=A0 583.21 GB=A0=A0=A0=A0=A0=A0 33.33%=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 141784319550391026443072753096570088105 = =A0 =A0


[user@node]$ ./checkinfo.sh =A0=A0=
*************** x.x.x.4
Token=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : 8507059= 1730234615865843651857942052863
Gossip active=A0=A0=A0 : true
Thrift = active=A0=A0=A0 : true
Load=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : 587.15= GB
Generation No=A0=A0=A0 : 1354074048
Uptime (seconds) : 36957
Heap Memory (MB) : 2027.29 / 3948.00
Data Center=A0=A0=A0=A0=A0 : DC1Rack=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : r2
Exceptions=A0=A0=A0=A0=A0= =A0 : 0

Key Cache=A0=A0=A0=A0=A0=A0=A0 : size 0 (bytes), capacity 0= (bytes), 0 hits, 0 requests, NaN recent hit rate, 14400 save period in sec= onds
Row Cache=A0=A0=A0=A0=A0=A0=A0 : size 1072651974 (bytes), capacity 10= 73741824 (bytes), 0 hits, 2576 requests, NaN recent hit rate, 0 save period= in seconds

*************** x.x.x.6
= Token=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : 1417843195503910264430727530965700= 88105
Gossip active=A0=A0=A0 : true
Thrift active=A0=A0=A0 : true
Load=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 := 583.21 GB
Generation No=A0=A0=A0 : 1354074461
Uptime (seconds) : 365= 35
Heap Memory (MB) : 828.71 / 3948.00
Data Center=A0=A0=A0=A0=A0 : D= C1
Rack=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : r2
Exceptions=A0=A0=A0= =A0=A0=A0 : 0

Key Cache=A0=A0=A0=A0=A0=A0=A0 : size 0 (bytes), capacity 0 (bytes), 0 = hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
Row Cache=A0=A0=A0=A0=A0=A0=A0 : size 1072602906 (bytes), capacity 107374= 1824 (bytes), 0 hits, 3194 requests, NaN recent hit rate, 0 save period in = seconds

--001636c5bb5b0a3ef204cfb8a960--