Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1ED81D800 for ; Tue, 18 Dec 2012 17:27:41 +0000 (UTC) Received: (qmail 50582 invoked by uid 500); 18 Dec 2012 17:27:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50549 invoked by uid 500); 18 Dec 2012 17:27:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50527 invoked by uid 99); 18 Dec 2012 17:27:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 17:27:37 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arodrime@gmail.com designates 209.85.220.173 as permitted sender) Received: from [209.85.220.173] (HELO mail-vc0-f173.google.com) (209.85.220.173) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Dec 2012 17:27:30 +0000 Received: by mail-vc0-f173.google.com with SMTP id f13so1177736vcb.4 for ; Tue, 18 Dec 2012 09:27:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=Rm3ghOBm56JllsWTtYOZxD6BGgTgLRrIVV8tnfl+2KY=; b=q81960/8Yx/mK6E0moAo//iNRFf5+6J9b6JTY+RdNFcCBmwTtPlNomZNrlGvflaG9s mVXA7EhnCwcND2fGlKm1M9q4j5oe/4jXIUDJs3IJFvT1rXwg0/L2oNGJriEnkxGQd+k9 /TxWBo9kw5UTgnNB7+pkbNZDzkD6sO6c32D7IKyGX2lCXHbsCGTfAFUpO2R4ruENL5kr v2FmtPAg8oMVUlr54RDSheRkD60FxOXtutVbkROtgirjsABiQbLsz8SG3ZOuUWYFLd0y xlZaDac4oQeGWmjlejxzHpLkXBtxT8l7JgL64+RfeWg1LABnux08BdyOSjpjJqUT+vkx +3tA== Received: by 10.220.153.201 with SMTP id l9mr4227954vcw.33.1355851629400; Tue, 18 Dec 2012 09:27:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.249.3 with HTTP; Tue, 18 Dec 2012 09:26:49 -0800 (PST) From: Alain RODRIGUEZ Date: Tue, 18 Dec 2012 18:26:49 +0100 Message-ID: Subject: High disk read throughput on only one node. To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04339ce4bd121304d123ce10 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04339ce4bd121304d123ce10 Content-Type: text/plain; charset=ISO-8859-1 Hi, I am experimenting a strange issue in my C* 1.1.6, 3 node, RF=3 cluster. root@ip-10-64-177-38:~# nodetool ring Note: Ownership information does not include topology, please specify a keyspace. Address DC Rack Status State Load Owns Token 141784319550391032739561396922763706368 10.64.167.32 eu-west 1b Up Normal 178.37 GB 33.33% 28356863910078203714492389662765613056 10.250.202.154 eu-west 1b Up Normal 174.93 GB 33.33% 85070591730234615865843651857942052863 10.64.177.38 eu-west 1b Up Normal 167.13 GB 33.33% 141784319550391032739561396922763706368 root@ip-10-64-177-38:~# nodetool ring cassa_teads Address DC Rack Status State Load Effective-Ownership Token 141784319550391032739561396922763706368 10.64.167.32 eu-west 1b Up Normal 178.37 GB 100.00% 28356863910078203714492389662765613056 10.250.202.154 eu-west 1b Up Normal 174.93 GB 100.00% 85070591730234615865843651857942052863 10.64.177.38 eu-west 1b Up Normal 167.13 GB 100.00% 141784319550391032739561396922763706368 My cluster is well balanced, all the nodes have an identical configuration, but yet I have a lot of disk reads on one of them as you can see in these screenshots: Datastax OpsCenter : http://img4.imageshack.us/img4/2528/datastaxopscenterheighr.png or AWS console : http://img59.imageshack.us/img59/5223/ec2managementconsole.png I have tried to see what is read from any nodes with "inotifywatch -r -t300 /raid0 > inotifywatch5min" and get the following result: root@ip-10-64-177-38:~# cat inotifywatch5min total access close_nowrite open filename 234580 113280 60691 60609 /raid0/cassandra/data/cassa_teads/data_viewer/ 56013 27108 14454 14451 /raid0/cassandra/data/cassa_teads/data_ip_viewer/ 30748 14998 7884 7866 /raid0/cassandra/data/cassa_teads/algo_ad_newcapping/ 301 147 76 78 /raid0/cassandra/data/cassa_teads/data_transac/ 191 95 48 48 /raid0/cassandra/data/cassa_teads/data_cust_website_viewer/ 6 0 3 3 /raid0/cassandra/ 2 0 1 1 /raid0/cassandra/data/ 2 0 1 1 /raid0/cassandra/commitlog/ 2 0 1 1 /raid0/cassandra/saved_caches/ root@ip-10-250-202-154:~# cat inotifywatch5min total access modify close_write close_nowrite open moved_from moved_to create delete filename 307378 115456 77706 12 57119 57035 /raid0/cassandra/data/cassa_teads/data_viewer/ 55395 26878 0 0 14259 14258 /raid0/cassandra/data/cassa_teads/data_ip_viewer/ 36155 17653 0 0 9256 9246 /raid0/cassandra/data/cassa_teads/algo_ad_newcapping/ 7377 188 7153 6 4 11 /raid0/cassandra/data/cassa_teads/data_action/ 401 0 364 6 4 12 /raid0/cassandra/data/cassa_teads/stats_ad_uv/ 244 120 0 0 62 62 /raid0/cassandra/data/cassa_teads/data_transac/ 160 76 0 0 42 42 /raid0/cassandra/data/cassa_teads/data_cust_website_viewer/ 26 0 0 0 13 13 /raid0/cassandra/data/cassa_teads/ 12 0 2 2 1 3 /raid0/cassandra/commitlog/ 6 0 0 0 3 3 /raid0/cassandra/ 2 0 0 0 1 1 /raid0/cassandra/data/ 2 0 0 0 1 1 /raid0/cassandra/saved_caches/ root@ip-10-64-167-32:~# cat inotifywatch5min total access modify close_write close_nowrite open filename 218489 105379 0 0 56615 56495 /raid0/cassandra/data/cassa_teads/data_viewer/ 61553 29733 0 0 15911 15909 /raid0/cassandra/data/cassa_teads/data_ip_viewer/ 33262 16214 0 0 8528 8520 /raid0/cassandra/data/cassa_teads/algo_ad_newcapping/ 407 0 370 6 4 12 /raid0/cassandra/data/cassa_teads/stats_ad_uv/ 280 134 0 0 73 73 /raid0/cassandra/data/cassa_teads/data_transac/ 163 79 0 0 42 42 /raid0/cassandra/data/cassa_teads/data_cust_website_viewer/ 17 0 3 3 1 4 /raid0/cassandra/commitlog/ 8 0 0 0 4 4 /raid0/cassandra/data/cassa_teads/ 6 0 0 0 3 3 /raid0/cassandra/ 2 0 0 0 1 1 /raid0/cassandra/data/ 2 0 0 0 1 1 /raid0/cassandra/saved_caches/ I see no reason to have a md0 disk read throughput up to 8 time bigger on the .38 node... root@ip-10-64-167-32:~# iostat -mx 5 100 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util xvdap1 0.00 0.40 0.20 0.40 0.00 0.00 18.67 0.00 6.67 6.67 0.40 xvdb 3.60 0.00 471.60 0.00 13.48 0.00 58.53 33.31 69.58 1.79 84.20 xvdc 3.40 0.00 474.40 0.00 13.76 0.00 59.42 30.06 62.63 1.72 81.60 xvdd 2.60 0.00 478.20 0.00 13.69 0.00 58.61 28.74 61.74 1.76 84.40 xvde 3.40 0.00 472.40 0.00 13.34 0.00 57.82 23.08 47.96 1.74 82.40 md0 0.00 0.00 1954.20 0.00 55.57 0.00 58.24 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 11.95 11.88 11.54 48.33 0.20 16.10 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util xvdap1 0.00 2.40 0.40 1.00 0.00 0.01 25.14 0.15 110.00 55.71 7.80 xvdb 3.40 0.00 481.60 63.20 13.94 2.59 62.13 33.79 62.86 1.56 84.80 xvdc 3.60 0.00 469.20 64.40 13.43 2.68 61.81 40.35 76.23 1.54 82.40 xvdd 6.60 0.20 480.80 64.40 13.93 2.65 62.28 45.99 84.72 1.63 88.60 xvde 6.60 0.00 482.00 65.60 14.20 2.68 63.14 48.76 89.77 1.61 88.00 md0 0.00 0.00 1910.80 257.80 54.92 10.60 61.88 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 12.41 12.82 10.93 36.15 0.21 27.48 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util xvdap1 0.00 0.60 2.40 0.60 0.03 0.00 23.47 0.18 58.67 24.67 7.40 xvdb 3.80 0.00 429.20 4.80 12.41 0.20 59.48 16.44 37.94 1.70 73.80 xvdc 3.00 0.00 432.60 4.80 12.47 0.20 59.33 20.74 47.44 1.71 75.00 xvdd 2.20 0.00 439.60 4.80 12.56 0.20 58.80 17.57 39.44 1.78 79.20 xvde 2.00 0.00 449.20 3.60 12.84 0.15 58.73 15.96 35.29 1.68 76.00 md0 0.00 0.00 1744.40 18.00 49.63 0.75 58.55 0.00 0.00 0.00 0.00 Did anyone experiments something similar ? How can I dig further to see what is wrong ? Any idea on what on C* can be responsible of this king of targeted reads ? Alain --f46d04339ce4bd121304d123ce10 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

I am experimenting a strange = issue in my C* 1.1.6, 3 node, RF=3D3 cluster.

root@ip-10-64-177-38:~# nodetool ring
Note: Ownershi= p information does not include topology, please specify a keyspace.
Address =A0 =A0 =A0 =A0 DC =A0 =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Stat= us State =A0 Load =A0 =A0 =A0 =A0 =A0 =A0Owns =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0Token
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A014178431955039103273956= 1396922763706368
10.64.167.32 =A0 =A0eu-west =A0 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 N= ormal =A0178.37 GB =A0 =A0 =A0 33.33% =A0 =A0 =A0 =A0 =A0 =A0 =A02835686391= 0078203714492389662765613056
10.250.202.154 =A0eu-west =A0 =A0 1b= =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A0174.93 GB =A0 =A0 =A0 33.33% =A0 = =A0 =A0 =A0 =A0 =A0 =A085070591730234615865843651857942052863
10.64.177.38 =A0 =A0eu-west =A0 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 N= ormal =A0167.13 GB =A0 =A0 =A0 33.33% =A0 =A0 =A0 =A0 =A0 =A0 =A01417843195= 50391032739561396922763706368

root@ip-10-64-177-38= :~# nodetool ring cassa_teads
Address =A0 =A0 =A0 =A0 DC =A0 =A0 =A0 =A0 =A0Rack =A0 =A0 =A0 =A0Status St= ate =A0 Load =A0 =A0 =A0 =A0 =A0 =A0Effective-Ownership Token
=A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0141784319550391032739561396922763706368
10.64.167.32 =A0 =A0eu-west =A0 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 N= ormal =A0178.37 GB =A0 =A0 =A0 100.00% =A0 =A0 =A0 =A0 =A0 =A0 283568639100= 78203714492389662765613056
10.250.202.154 =A0eu-west =A0 =A0 1b = =A0 =A0 =A0 =A0 =A0Up =A0 =A0 Normal =A0174.93 GB =A0 =A0 =A0 100.00% =A0 = =A0 =A0 =A0 =A0 =A0 85070591730234615865843651857942052863
10.64.177.38 =A0 =A0eu-west =A0 =A0 1b =A0 =A0 =A0 =A0 =A0Up =A0 =A0 N= ormal =A0167.13 GB =A0 =A0 =A0 100.00% =A0 =A0 =A0 =A0 =A0 =A0 141784319550= 391032739561396922763706368

My c= luster is well balanced, all the nodes have an identical configuration, but= yet I have a lot of disk reads on one of them as you can see in these scre= enshots:

or
AWS console :=A0http://img59.imageshack.us/img59/5223/ec2managementconsole.= png

I have tried to see what is re= ad from any nodes with "inotifywatch -r -t300 /raid0 > inotifywatch= 5min" and get the following result:

root@ip-10-64-177-38:~# cat inotifywat= ch5min
total =A0 =A0 =A0 access =A0close_nowrite =A0open =A0 =A0 = =A0filename
234580 =A0 113280 =A060691 =A0 =A0 =A0 =A0 =A0 =A0 60= 609 =A0 =A0/raid0/cassandra/data/cassa_teads/data_viewer/
56013 =A0 =A0 =A027108 =A0 14454 =A0 =A0 =A0 =A0 =A0 =A014451 =A0 =A0 = /raid0/cassandra/data/cassa_teads/data_ip_viewer/
30748 =A0 =A0 = =A014998 =A0 7884 =A0 =A0 =A0 =A0 =A0 =A0 =A07866 =A0 =A0 =A0/raid0/cassand= ra/data/cassa_teads/algo_ad_newcapping/
301 =A0 =A0 =A0 =A0 =A0147 =A0 =A0 =A076 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A078 =A0 =A0 =A0 =A0 /raid0/cassandra/data/cassa_teads/data_transac/
=
191 =A0 =A0 =A0 =A0 =A0 95 =A0 =A0 =A0 48 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A048 =A0 =A0=A0 =A0=A0=A0/raid0/cassandra/data/cassa_teads/data_cust_w= ebsite_viewer/
6 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A03 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A03 =A0 =A0=A0=A0 =A0=A0 =A0/raid0/cassandra/
2 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A01 =A0 =A0=A0=A0 =A0 =A0=A0/raid0/cassandra/data/
2 =A0= =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A01 =A0 =A0=A0=A0 =A0 =A0=A0/raid0/cassandra/commitlog/
2 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A01 =A0 =A0=A0 =A0=A0 =A0=A0/raid0/cassandra/saved_caches/=


root@ip-10-250-202-154:~# cat= inotifywatch5min
total =A0 =A0 =A0 access =A0modify =A0close_wri= te =A0close_nowrite =A0open =A0moved_from =A0moved_to =A0create =A0delete = =A0filename
307378 =A0115456 =A077706 =A0 12 =A0 =A0=A0=A0 =A0 =A0=A0 =A0 =A0=A057= 119 =A0=A0=A0 =A0 =A0=A0 =A0 =A057035 =A0 =A0 =A0 =A0/raid0/cassandra/data/= cassa_teads/data_viewer/
55395 =A0 =A026878 =A0 0 =A0 =A0 =A0 =A0= =A0 0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A014259 =A0 =A0 =A0=A0=A0 =A0 =A0= =A014258 =A0 =A0 =A0 /raid0/cassandra/data/cassa_teads/data_ip_viewer/
36155 =A0 =A017653 =A0 0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0=A0=A0 =A0 =A0= =A0 =A0 =A0 =A09256 =A0 =A0 =A0 =A0=A0 =A0 =A0=A0 9246 =A0 =A0 =A0 =A0/raid= 0/cassandra/data/cassa_teads/algo_ad_newcapping/
7377 =A0 =A0 =A0= 188 =A0 =A0 =A07153 =A0 =A0 =A06 =A0 =A0 =A0 =A0 =A0=A0 =A0 =A0=A0=A04 =A0 = =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A0 =A0 11 =A0 =A0 =A0 =A0 =A0 /raid0/cassandr= a/data/cassa_teads/data_action/
401 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 364 =A0 =A0 =A0 =A06 =A0 =A0 =A0 = =A0=A0=A0 =A0 =A0=A0 =A04 =A0 =A0 =A0 =A0 =A0=A0 =A0 =A0=A0 =A0 =A012 =A0 = =A0=A0 =A0 =A0=A0 /raid0/cassandra/data/cassa_teads/stats_ad_uv/
= 244 =A0 =A0 =A0 =A0120 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 = =A0=A0 =A0 =A0=A0=A062 =A0 =A0 =A0=A0 =A0 =A0=A0 =A0 =A0 =A0 62 =A0 =A0=A0 = =A0 =A0=A0/raid0/cassandra/data/cassa_teads/data_transac/
160 =A0 =A0 =A0 =A076 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0=A0 =A0 =A0=A0=A042 =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A0 =A0 =A042 = =A0 =A0=A0 =A0 =A0/raid0/cassandra/data/cassa_teads/data_cust_website_viewe= r/
26 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 = 0 =A0 =A0 =A0 =A0 =A0 =A0=A0 =A0 =A013 =A0 =A0 =A0 =A0 =A0=A0 =A0 =A0=A0 = =A0 13 =A0 =A0=A0 =A0 =A0=A0/raid0/cassandra/data/cassa_teads/
12 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 2 =A0 =A0 =A0 =A0 =A0 2 =A0 = =A0 =A0 =A0 =A0=A0 =A0 =A0=A0=A01 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A0 =A0 = =A03 =A0 =A0=A0=A0 =A0 =A0=A0/raid0/cassandra/commitlog/
6 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0=A0=A0 =A0 =A0=A03 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A0 =A0 =A03 =A0= =A0=A0=A0 =A0 =A0/raid0/cassandra/
2 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 0 =A0= =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A01 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A0=A0 =A0= =A01 =A0 =A0=A0=A0 =A0 =A0/raid0/cassandra/data/
2 =A0 =A0 =A0 = =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0=A0= =A0 =A0 =A0=A01 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 =A0 =A0 =A01=A0 =A0 =A0=A0 = =A0 =A0/raid0/cassandra/saved_caches/


root@ip-10-64-167-32:~# cat inotif= ywatch5min
total =A0 =A0 =A0 access =A0modify =A0close_w= rite =A0close_nowrite =A0open =A0 =A0filename
218489 =A0 105379 = =A00 =A0 =A0=A0=A0=A0 =A0 =A0=A00 =A0 =A0 =A0=A0=A0 =A0 =A0=A0=A0 =A0 =A056= 615 =A0 =A0 =A0 =A0 56495 =A0/raid0/cassandra/data/cassa_teads/data_viewer/=
61553 =A0=A0=A0 29733 =A0 0 =A0 =A0 =A0=A0=A0=A0 =A0 =A00 =A0 =A0 =A0 = =A0 =A0=A0=A0 =A0 =A0=A0=A015911 =A0 =A0 =A0 =A015909 =A0/raid0/cassandra/d= ata/cassa_teads/data_ip_viewer/
33262 =A0 =A0 16214 =A0 0 =A0 =A0= =A0=A0=A0=A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0=A0=A0=A0 =A0 =A0=A08528 =A0 =A0 = =A0 =A0 =A08520 =A0 =A0/raid0/cassandra/data/cassa_teads/algo_ad_newcapping= /
407 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 370 =A0 =A0=A0=A0=A0 =A0 6 =A0 = =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A04 =A0 =A0 =A0 =A0 =A0 =A0 =A0 12 =A0 =A0= =A0=A0 =A0/raid0/cassandra/data/cassa_teads/stats_ad_uv/
280 =A0 = =A0 =A0 =A0 134 =A0 =A0 =A00 =A0 =A0 =A0=A0=A0=A0 =A0 =A00 =A0 =A0 =A0 =A0 = =A0 =A0=A0=A0 =A0 =A073 =A0 =A0 =A0 =A0 =A0 =A0 =A073 =A0 =A0 =A0 /raid0/ca= ssandra/data/cassa_teads/data_transac/
163 =A0 =A0 =A0 =A0 =A079 =A0 =A0 =A0 0 =A0 =A0 =A0=A0=A0=A0 =A0 =A00 = =A0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A042 =A0 =A0 =A0 =A0 =A0 =A0 =A042 =A0 = =A0 =A0 /raid0/cassandra/data/cassa_teads/data_cust_website_viewer/
17 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A03 =A0 =A0 =A0=A0=A0=A0 =A0 = =A03 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= 4 =A0 =A0 =A0 =A0/raid0/cassandra/commitlog/
8 =A0 =A0 =A0=A0=A0 =A0 =A0=A00 =A0 =A0 =A0 =A00 =A0 =A0 =A0=A0=A0=A0 = =A0 =A0=A00 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A04 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 4 =A0 =A0 =A0 =A0/raid0/cassandra/data/cassa_teads/
6 =A0= =A0 =A0=A0=A0 =A0 =A0=A00 =A0 =A0 =A0 =A00 =A0 =A0 =A0=A0=A0=A0 =A0 =A0=A0= 0 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A03 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 3 = =A0 =A0 =A0 =A0/raid0/cassandra/
2 =A0 =A0 =A0=A0=A0 =A0 =A0=A00 =A0 =A0 =A0 =A00 =A0 =A0 =A0=A0=A0=A0 = =A0 =A0=A00 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 1 =A0 =A0 =A0 =A0/raid0/cassandra/data/
2 =A0 =A0 =A0=A0= =A0 =A0 =A0=A00 =A0 =A0 =A0 =A00 =A0 =A0 =A0=A0=A0=A0 =A0 =A0=A00 =A0 =A0 = =A0 =A0 =A0 =A0=A0=A0 =A0 =A01 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0 = =A0 =A0/raid0/cassandra/saved_caches/

I see no reason to have a md0 d= isk read throughput up to 8 time bigger on the .38 node...
=
root@ip-10-64-167-32:~#=A0iostat -mx 5 100

Device: =A0 =A0 =A0 =A0 rrqm/s = =A0 wrqm/s =A0 =A0 r/s =A0 =A0 w/s =A0 =A0rMB/s =A0 =A0wMB/s avgrq-sz avgqu= -sz =A0 await =A0svctm =A0%util
xvdap1 =A0 =A0 =A0 =A0 =A0 =A00.0= 0 =A0 =A0 0.40 =A0 =A00.20 =A0 =A00.40 =A0 =A0 0.00 =A0 =A0 0.00 =A0 =A018.= 67 =A0 =A0 0.00 =A0 =A06.67 =A0 6.67 =A0 0.40
xvdb =A0 =A0 =A0 =A0 =A0 =A0 =A03.60 =A0 =A0 0.00 =A0471.60 =A0 =A00.0= 0 =A0 =A013.48 =A0 =A0 0.00 =A0 =A058.53 =A0 =A033.31 =A0 69.58 =A0 1.79 = =A084.20
xvdc =A0 =A0 =A0 =A0 =A0 =A0 =A03.40 =A0 =A0 0.00 =A0474= .40 =A0 =A00.00 =A0 =A013.76 =A0 =A0 0.00 =A0 =A059.42 =A0 =A030.06 =A0 62.= 63 =A0 1.72 =A081.60
xvdd =A0 =A0 =A0 =A0 =A0 =A0 =A02.60 =A0 =A0 0.00 =A0478.20 =A0 =A00.0= 0 =A0 =A013.69 =A0 =A0 0.00 =A0 =A058.61 =A0 =A028.74 =A0 61.74 =A0 1.76 = =A084.40
xvde =A0 =A0 =A0 =A0 =A0 =A0 =A03.40 =A0 =A0 0.00 =A0472= .40 =A0 =A00.00 =A0 =A013.34 =A0 =A0 0.00 =A0 =A057.82 =A0 =A023.08 =A0 47.= 96 =A0 1.74 =A082.40
md0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00 =A0 =A0 0.00 1954.20 =A0 =A00.00 = =A0 =A055.57 =A0 =A0 0.00 =A0 =A058.24 =A0 =A0 0.00 =A0 =A00.00 =A0 0.00 = =A0 0.00

avg-cpu: =A0%user =A0 %nice %system %iowa= it =A0%steal =A0 %idle
=A0 =A0 =A0 =A0 =A0 11.95 =A0 11.88 =A0 11= .54 =A0 48.33 =A0 =A00.20 =A0 16.10

Device: =A0 =A0 =A0 =A0 rrqm/s =A0 wrqm/s =A0 =A0 r/s = =A0 =A0 w/s =A0 =A0rMB/s =A0 =A0wMB/s avgrq-sz avgqu-sz =A0 await =A0svctm = =A0%util
xvdap1 =A0 =A0 =A0 =A0 =A0 =A00.00 =A0 =A0 2.40 =A0 =A00= .40 =A0 =A01.00 =A0 =A0 0.00 =A0 =A0 0.01 =A0 =A025.14 =A0 =A0 0.15 =A0110.= 00 =A055.71 =A0 7.80
xvdb =A0 =A0 =A0 =A0 =A0 =A0 =A03.40 =A0 =A0 0.00 =A0481.60 =A0 63.20 = =A0 =A013.94 =A0 =A0 2.59 =A0 =A062.13 =A0 =A033.79 =A0 62.86 =A0 1.56 =A08= 4.80
xvdc =A0 =A0 =A0 =A0 =A0 =A0 =A03.60 =A0 =A0 0.00 =A0469.20 = =A0 64.40 =A0 =A013.43 =A0 =A0 2.68 =A0 =A061.81 =A0 =A040.35 =A0 76.23 =A0= 1.54 =A082.40
xvdd =A0 =A0 =A0 =A0 =A0 =A0 =A06.60 =A0 =A0 0.20 =A0480.80 =A0 64.40 = =A0 =A013.93 =A0 =A0 2.65 =A0 =A062.28 =A0 =A045.99 =A0 84.72 =A0 1.63 =A08= 8.60
xvde =A0 =A0 =A0 =A0 =A0 =A0 =A06.60 =A0 =A0 0.00 =A0482.00 = =A0 65.60 =A0 =A014.20 =A0 =A0 2.68 =A0 =A063.14 =A0 =A048.76 =A0 89.77 =A0= 1.61 =A088.00
md0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00 =A0 =A0 0.00 1910.80 =A0257.80 = =A0 =A054.92 =A0 =A010.60 =A0 =A061.88 =A0 =A0 0.00 =A0 =A00.00 =A0 0.00 = =A0 0.00

avg-cpu: =A0%user =A0 %nice %system %iowa= it =A0%steal =A0 %idle
=A0 =A0 =A0 =A0 =A0 12.41 =A0 12.82 =A0 10= .93 =A0 36.15 =A0 =A00.21 =A0 27.48

Device: =A0 =A0 =A0 =A0 rrqm/s =A0 wrqm/s =A0 =A0 r/s = =A0 =A0 w/s =A0 =A0rMB/s =A0 =A0wMB/s avgrq-sz avgqu-sz =A0 await =A0svctm = =A0%util
xvdap1 =A0 =A0 =A0 =A0 =A0 =A00.00 =A0 =A0 0.60 =A0 =A02= .40 =A0 =A00.60 =A0 =A0 0.03 =A0 =A0 0.00 =A0 =A023.47 =A0 =A0 0.18 =A0 58.= 67 =A024.67 =A0 7.40
xvdb =A0 =A0 =A0 =A0 =A0 =A0 =A03.80 =A0 =A0 0.00 =A0429.20 =A0 =A04.8= 0 =A0 =A012.41 =A0 =A0 0.20 =A0 =A059.48 =A0 =A016.44 =A0 37.94 =A0 1.70 = =A073.80
xvdc =A0 =A0 =A0 =A0 =A0 =A0 =A03.00 =A0 =A0 0.00 =A0432= .60 =A0 =A04.80 =A0 =A012.47 =A0 =A0 0.20 =A0 =A059.33 =A0 =A020.74 =A0 47.= 44 =A0 1.71 =A075.00
xvdd =A0 =A0 =A0 =A0 =A0 =A0 =A02.20 =A0 =A0 0.00 =A0439.60 =A0 =A04.8= 0 =A0 =A012.56 =A0 =A0 0.20 =A0 =A058.80 =A0 =A017.57 =A0 39.44 =A0 1.78 = =A079.20
xvde =A0 =A0 =A0 =A0 =A0 =A0 =A02.00 =A0 =A0 0.00 =A0449= .20 =A0 =A03.60 =A0 =A012.84 =A0 =A0 0.15 =A0 =A058.73 =A0 =A015.96 =A0 35.= 29 =A0 1.68 =A076.00
md0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00 =A0 =A0 0.00 1744.40 =A0 18.00 = =A0 =A049.63 =A0 =A0 0.75 =A0 =A058.55 =A0 =A0 0.00 =A0 =A00.00 =A0 0.00 = =A0 0.00

Did anyone experiments = something similar ?

How can I dig further to see what is wrong ?
Any idea on what on C* can be responsible of this king of= targeted reads ?

Alain
--f46d04339ce4bd121304d123ce10--