Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A74549FF5 for ; Wed, 10 Oct 2012 20:11:41 +0000 (UTC) Received: (qmail 14045 invoked by uid 500); 10 Oct 2012 20:11:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 14007 invoked by uid 500); 10 Oct 2012 20:11:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13998 invoked by uid 99); 10 Oct 2012 20:11:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Oct 2012 20:11:39 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 74.125.82.172 is neither permitted nor denied by domain of tamar@tok-media.com) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Oct 2012 20:11:32 +0000 Received: by mail-we0-f172.google.com with SMTP id u46so641901wey.31 for ; Wed, 10 Oct 2012 13:11:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=dzLYFgJmMo0wXLv6Nkmk5WV+9s8QtW30F+aRHu+QGTE=; b=XGiQh3kJNAf9GALamipZtvLil2BdmLwFttUT/RVvJtf+g1Mfh99SkXSBZW2/uuMeZW cLNe5zjiCFVyexdQoIxg5C6RTyAbC6N+h7cvbJYnHrZBPeJUMWyWyjDyAfmUysRAF9W9 L4oXNctBUK9udIT05PMmuk3SqR7YqWGUpsNIMtidziLapAkdpzkZ1fbDSfaVXI4w1meZ dGC54vm1fKZpxsLL5e7uVJQg8Vk66AhR5fsF1/KGsa0mVhM0zvxmvsGh5gIvXmf9AZlD BUXga4e9XmcyJ4v+JiLrp3O0XfTzLixAEF0kZyYSC94Qxw7RZ0ahU5WiPnbC3PAD9eX0 h/nQ== Received: by 10.180.76.11 with SMTP id g11mr14883734wiw.13.1349899872139; Wed, 10 Oct 2012 13:11:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.5.131 with HTTP; Wed, 10 Oct 2012 13:10:50 -0700 (PDT) In-Reply-To: <19C8179E-1594-4190-AB19-4F57328B03AF@thelastpickle.com> References: <19C8179E-1594-4190-AB19-4F57328B03AF@thelastpickle.com> From: Tamar Fraenkel Date: Wed, 10 Oct 2012 22:10:50 +0200 Message-ID: Subject: Re: READ messages dropped To: user@cassandra.apache.org Content-Type: multipart/related; boundary=f46d043893d55c785f04cbba0e10 X-Gm-Message-State: ALoCoQkJv7fnkbFmDsnQhQhHkD8nLrscg8BW9F5t8sWwS7W9HCx99mBm2n5ymqMJ2RntAP5Gu93n --f46d043893d55c785f04cbba0e10 Content-Type: multipart/alternative; boundary=f46d043893d55c785a04cbba0e0f --f46d043893d55c785a04cbba0e0f Content-Type: text/plain; charset=ISO-8859-1 Hi! Thanks for the answer. I don't see much change in the load this Cassandra cluster is under, so why is the sudden surge of such messages? What I did noticed while looking at the logs (which are also running OpsCenter), is that there is some correlation between the dropped reads and flushes of OpsCenter column families to disk and or compactions. What are the rollups CFs? why is there so much traffic in them? Thanks, *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] tamar@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Wed, Oct 10, 2012 at 1:00 AM, aaron morton wrote: > or how to solve it? > > Simple solution is move to m1.xlarge :) > > In the last 3 days I see many messages of "READ messages dropped in last > 5000ms" on one of my 3 nodes cluster. > > The node is not able to keep up with the load. > > Possible causes include excessive GC, aggressive compaction, or simply too > many requests. > > it also a good idea to take a look at iostat to see if the disk is keeping > up. > > Hope that helps > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 9/10/2012, at 9:08 AM, Tamar Fraenkel wrote: > > Hi! > In the last 3 days I see many messages of "READ messages dropped in last > 5000ms" on one of my 3 nodes cluster. > I see no errors in the log. > There are also messages of "Finished hinted handoff of 0 rows to endpoint" > but I had those for a while now, so I don't know if they are related. > I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large > instances. Rep factor 3 (Quorum read and write) > > Does anyone have a clue what I should be looking for, or how to solve it? > Thanks, > > *Tamar Fraenkel * > Senior Software Engineer, TOK Media > > > > > tamar@tok-media.com > Tel: +972 2 6409736 > Mob: +972 54 8356490 > Fax: +972 2 5612956 > > > > > --f46d043893d55c785a04cbba0e0f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi!
Thanks for the answer.
I don't see much chan= ge in the load this Cassandra cluster is under, so why is the sudden surge = of such messages?
What I did noticed while looking at the logs (which ar= e also running OpsCenter), is that there is some correlation between the dr= opped reads and flushes of OpsCenter column families to disk and or compact= ions. What are the rollups CFs? why is there so much traffic in them?
Thanks,
Tamar Fraenke= l=A0
Senior Software Engineer, TOK Media=A0

= 3D"Inline

tamar@tok-media.com<= /a>
Tel:=A0=A0=A0+972 2 6409736=A0
Mob:=A0=A0<= a value=3D"+972548356490">+972 54 8356490=A0
Fax:=A0=A0=A0+972 2 5612956=A0





On Wed, Oct 10, 2012 at 1:00 AM, aaron m= orton <aaron@thelastpickle.com> wrote:
or how to solve it?
Simple so= lution is move to m1.xlarge :)

In the last 3 days I see many messages of "READ messa= ges dropped in last 5000ms" on one of my 3 nodes cluster.
The node is not able to keep up with the load.=A0

Possible causes include excessive GC, aggressive compaction,= or simply too many requests.

it also a good idea = to take a look at iostat to see if the disk is keeping up.=A0

Hope that helps=A0
=A0
<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 9/10/2012, at 9:08 AM, Tamar Fraenkel &l= t;tamar@tok-media.= com> wrote:

Hi!
In the last 3 days I see many messages of "READ messages droppe= d in last 5000ms" on one of my 3 nodes cluster.
I see no errors in = the log.
There are also messages of "Finished hinted handoff of 0 r= ows to endpoint" but I had those for a while now, so I don't know = if they are related.
I am running Cassandra 1.0.8 on a 3 node cluster on EC2 m1.large instances.= Rep factor 3 (Quorum read and write)

Does anyone have a clue what I= should be looking for, or how to solve it?
Thanks,

Tamar Fraenkel= =A0
Senior Software Engineer, TOK Media=A0

<tokLogo.png>


tamar@tok-media.com Tel:=A0=A0=A0+972 2 6409736=A0
Mob:=A0=A0<= a value=3D"+972548356490">+972 54 8356490=A0
Fax:=A0=A0=A0+972 2 5612956=A0





--f46d043893d55c785a04cbba0e0f-- --f46d043893d55c785f04cbba0e10 Content-Type: image/png; name="tokLogo.png" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: ii_135b91fb888fa9ff iVBORw0KGgoAAAANSUhEUgAAAF0AAAAnCAYAAABtyERkAAAABGdBTUEAALGPC/xhBQAACkNpQ0NQ SUNDIFByb2ZpbGUAAHgBnZZ3VFNZE8Dvey+90BJCkRJ6DU1KAJESepFeRSUkAUIJGBKwV0QFVxQV aYoiiyIuuLoUWSuiWFgUFLAvyCKgrIuriIplX/QcZf/Y/b6z88ec35s7c+/cmbnnPAAovoFCUSas AECGSCIO8/FgxsTGMfHdAAZEgAPWAHB52VlB4d4RABU/Lw4zG3WSsUygz/p1/xe4xfINYTI/m/5/ pcjLEkvQnULQkLl8QTYP5TyU03MlWTL7JMr0xDQZwxgZi9EEUVaVcfIXNv/s84XdZMzPEPFRH1nO WfwMvow7UN6SIxWgjASinJ8jFOSifBtl/XRphhDlNyjTMwTcbAAwFJldIuCloGyFMkUcEcZBeR4A BEryLE6cxRLBMjRPADiZWcvFwuQUCdOYZ8K0dnRkM30FuekCiYQVwuWlccV8JiczI4srWg7AlzvL ooCSrLZMtMj21o729iwbC7T8X+VfF796/TvIevvF42Xo555BjK5vtm+x32yZ1QCwp9Da7PhmSywD oGUTAKr3vtn0DwAgnwdA841Z92HI5iVFIslysrTMzc21EAp4FrKCfpX/6fDV859h1nkWsvO+1o7p KUjiStMlTFlReZnpmVIxMzuLyxMwWX8bYnTr/xw4K61ZeZiHCZIEYoEIPSoKnTKhKBltt4gvlAgz RUyh6J86/B/DZuUgwy9zjQKt5iOgL7EACjfoAPm9C2BoZIDE70dXoK99CyRGAdnLi9Ye/TL3KKPr n/XfFFyEfsLZwmSmzMwJi2DypOIcGaNvQqawgATkAR2oAS2gB4wBC9gAB+AM3IAX8AfBIALEgsWA B1JABhCDXLAKrAf5oBDsAHtAOagCNaAONIAToAWcBhfAZXAd3AR94D4YBCPgGZgEr8EMBEF4iArR IDVIGzKAzCAbiA3Nh7ygQCgMioUSoGRIBEmhVdBGqBAqhsqhg1Ad9CN0CroAXYV6oLvQEDQO/Qm9 gxGYAtNhTdgQtoTZsDscAEfAi+BkeCm8As6Dt8OlcDV8DG6GL8DX4T54EH4GTyEAISMMRAdhIWyE gwQjcUgSIkbWIAVICVKNNCBtSCdyCxlEJpC3GByGhmFiWBhnjC8mEsPDLMWswWzDlGOOYJoxHZhb mCHMJOYjlorVwJphnbB+2BhsMjYXm48twdZim7CXsH3YEexrHA7HwBnhHHC+uFhcKm4lbhtuH64R dx7XgxvGTeHxeDW8Gd4FH4zn4iX4fHwZ/hj+HL4XP4J/QyATtAk2BG9CHEFE2EAoIRwlnCX0EkYJ M0QFogHRiRhM5BOXE4uINcQ24g3iCHGGpEgyIrmQIkippPWkUlID6RLpAeklmUzWJTuSQ8lC8jpy Kfk4+Qp5iPyWokQxpXAo8RQpZTvlMOU85S7lJZVKNaS6UeOoEup2ah31IvUR9Y0cTc5Czk+OL7dW rkKuWa5X7rk8Ud5A3l1+sfwK+RL5k/I35CcUiAqGChwFrsIahQqFUwoDClOKNEVrxWDFDMVtikcV ryqOKeGVDJW8lPhKeUqHlC4qDdMQmh6NQ+PRNtJqaJdoI3Qc3YjuR0+lF9J/oHfTJ5WVlG2Vo5SX KVcon1EeZCAMQ4YfI51RxDjB6Ge8U9FUcVcRqGxVaVDpVZlWnaPqpipQLVBtVO1TfafGVPNSS1Pb qdai9lAdo26qHqqeq75f/ZL6xBz6HOc5vDkFc07MuacBa5hqhGms1Dik0aUxpaml6aOZpVmmeVFz Qouh5aaVqrVb66zWuDZNe762UHu39jntp0xlpjsznVnK7GBO6mjo+OpIdQ7qdOvM6BrpRupu0G3U fahH0mPrJent1mvXm9TX1g/SX6Vfr3/PgGjANkgx2GvQaTBtaGQYbbjZsMVwzEjVyM9ohVG90QNj qrGr8VLjauPbJjgTtkmayT6Tm6awqZ1pimmF6Q0z2MzeTGi2z6zHHGvuaC4yrzYfYFFY7qwcVj1r yIJhEWixwaLF4rmlvmWc5U7LTsuPVnZW6VY1Vvetlaz9rTdYt1n/aWNqw7OpsLk9lzrXe+7aua1z X9ia2Qps99vesaPZBdlttmu3+2DvYC+2b7Afd9B3SHCodBhg09kh7G3sK45YRw/HtY6nHd862TtJ nE44/eHMck5zPuo8Ns9onmBezbxhF10XrstBl8H5zPkJ8w/MH3TVceW6Vrs+dtNz47vVuo26m7in uh9zf+5h5SH2aPKY5jhxVnPOeyKePp4Fnt1eSl6RXuVej7x1vZO9670nfex8Vvqc98X6Bvju9B3w 0/Tj+dX5Tfo7+K/27wigBIQHlAc8DjQNFAe2BcFB/kG7gh4sMFggWtASDIL9gncFPwwxClka8nMo LjQktCL0SZh12KqwznBa+JLwo+GvIzwiiiLuRxpHSiPbo+Sj4qPqoqajPaOLowdjLGNWx1yPVY8V xrbG4eOi4mrjphZ6LdyzcCTeLj4/vn+R0aJli64uVl+cvvjMEvkl3CUnE7AJ0QlHE95zg7nV3KlE v8TKxEkeh7eX94zvxt/NHxe4CIoFo0kuScVJY8kuybuSx1NcU0pSJoQcYbnwRapvalXqdFpw2uG0 T+nR6Y0ZhIyEjFMiJVGaqCNTK3NZZk+WWVZ+1uBSp6V7lk6KA8S12VD2ouxWCR39meqSGks3SYdy 5udU5LzJjco9uUxxmWhZ13LT5VuXj67wXvH9SsxK3sr2VTqr1q8aWu2++uAaaE3imva1emvz1o6s 81l3ZD1pfdr6XzZYbSje8Gpj9Ma2PM28dXnDm3w21efL5YvzBzY7b67agtki3NK9de7Wsq0fC/gF 1wqtCksK32/jbbv2nfV3pd992p60vbvIvmj/DtwO0Y7+na47jxQrFq8oHt4VtKt5N3N3we5Xe5bs uVpiW1K1l7RXunewNLC0tUy/bEfZ+/KU8r4Kj4rGSo3KrZXT+/j7eve77W+o0qwqrHp3QHjgzkGf g83VhtUlh3CHcg49qYmq6fye/X1drXptYe2Hw6LDg0fCjnTUOdTVHdU4WlQP10vrx4/FH7v5g+cP rQ2shoONjMbC4+C49PjTHxN+7D8RcKL9JPtkw08GP1U20ZoKmqHm5c2TLSktg62xrT2n/E+1tzm3 Nf1s8fPh0zqnK84onyk6Szqbd/bTuRXnps5nnZ+4kHxhuH1J+/2LMRdvd4R2dF8KuHTlsvfli53u neeuuFw5fdXp6qlr7Gst1+2vN3fZdTX9YvdLU7d9d/MNhxutNx1vtvXM6znb69p74Zbnrcu3/W5f 71vQ19Mf2X9nIH5g8A7/ztjd9Lsv7uXcm7m/7gH2QcFDhYcljzQeVf9q8mvjoP3gmSHPoa7H4Y/v D/OGn/2W/dv7kbwn1Cclo9qjdWM2Y6fHvcdvPl34dORZ1rOZifzfFX+vfG78/Kc/3P7omoyZHHkh fvHpz20v1V4efmX7qn0qZOrR64zXM9MFb9TeHHnLftv5Lvrd6Ezue/z70g8mH9o+Bnx88Cnj06e/ AAOb8/zszueKAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAHmklEQVRoBeVaa0wUVxQ+uzPLLqj1lbRW IkKKYgNq0WpRq0mjEWJRqVaLGmoCFdpCpYhGLP6hlvpCWl+NmGATFYH+qPhqq9W0wSqgLYhPFCxI Q2L7oyjCPthX7xkY2MedYRaGWbAnmcy959x77plvzp69c+5R2QnBICLLn7UAqg6D2aAJg8jyblPZ 7ubAatma/wVzVSVYblaB7U4l2OquAfgQGxlyodV4qQn+rAqYKZGgDp4M7MSpwASFgHr4KCIcuKRC T3/8+G8oLi7uVyvnzZsL4eHhomvYDQYw37gBphMFYC09D6AhP0ICrkpDprGkLQB614vAF0Iudnos aGa9DZqps0TXcxWePn0G6uvrXdkQFBQES5YsduPTGFKw5DzdaDRAeXkFTYdsvNDQUFFdpp8vgiF7 O9ibGjtARqB7SZbKIsDL5D8DdKsyuF+AFFUIeF9wMBqN8NVXX0NjY6PgckuXLsYfqHfJ2tQELR+l gD7+Y7A/+ktWY2xN10Gfsxz0x3aD3WSQVberMgR8z57cHgGPjY31Lujm6mp4tup9sJ694PoMsvYt l/OgbUc8WP9pklUvr4wH/PbtOzzL7Y4ejoAjceHFbYQCjParZdD23lpJK7FRq4Cd9jq3a2FCOsMU 2cFYa++A3dAGtkf3wPJ7oaguzut3xoHf5mPAvOgvOtYToaeAo26vgI4hRZ+xVfTZ2HdiQbt8BWgm TxEcxwYGd8nspkyw1NwAc9k5Es/pL8D+rBH0BPihX54Dlda3a25fGgUFJ0Cqh/PrcKAHBgZCYWEB z6PeGxoaYMuWTKps+/ZsQB1SCHcorWkbwV5Pj99MVBT4paYCG9wNqBS9Kq2O7FYiuMvyYAUYC3eA rYlsM10IgTd8mw1+H37hIvG8W1RUBBcvXhKc6BhSHAcp/kdqOn8BbGV/ONrQ1dZuTocX9u/3GPAu BZ0NduIUGJJxBNi5Sa4iro87G3N1GVUmlYmAnzp1RnC4EOA4QVHQ0cuNufuohmoz04n30UGiTuiB iZ7vF7cJNJGbqCNNJXQ7qINdmH0BHFUpCnr7b1eoYUXzSaKsgDtipIteC0xIlCOLa+Mfq+VBtRu/ J8aVK1d77eG8bmVB/+VXfl2nu9+6BKe+nB30eN2KT6kqzbc8CzFVVVVw4MBBqi5kioUUx0mKgm45 9p3j2lwbvVw9cqQbX04GExAMav+ZbiptDbfceEIMBHzXrhwhsWTAUQG3exHUJKMAt4k00kwTz8fQ 5vSGx86MhvaTzrsZ632S35FANTU1oukBqR7OL6WYp9v1en5NpzsTON6p318dle/QXqt+8uSp6Ny2 NvqzCU1SDHQhA54HPu7V8Q9WKikGunr0aKpN1oZHVL7cTLuhVW6VTvqOHz8OmNaVQsqBPop+sGCu rJJiZ5/HWK6dddPBhES68XrLwBCUn58PmIvpiRQDHQ3RpKxzs8e8/zDYmpvd+HIyzDfLqSkB5tXZ Hi8zYsRwwTmYgykpKRGU8wJFQfeZ+ya/rtO9LSfXqS9nx24ygukk/etTE+a+jRRbOywslBxS5ALe hQhTA7jbESNFQdeEvwaqoHFu9liOFoP+UJ4bv68MBFx/YCPVy9XkVIkJmCB5CQQ6PX0D6HQ6SEgQ /5jbu3cfiO14FAVd5esLug3rqQ9qyt4DrbtzwC4hJlIVuDB5wK33f3KRdHTxGE8qBQQEdAGOc8aM eQlSUpIFpyPgBw8Kf7kqCjpaqYtZCkz0QqrB5m8OQ0t8PFjq6qhyqcz2ikvQmhkNQoBj9hErB6TS 2LEvcx7uOH7OnNkQEfGGI8upjfEdD7pppDjoaMTQ7M+pYQZltuskEXX3LjY9ItvTZmgvJ2BnrQZj fhLYWxqo83HH4rsyhSrzlJmUlAhif6yFhUWA5xCupFgawHFhNdk+Dis8yp2PCh1mOI7Htu1JM5jL S7kjO67YiBzXIVkbH4DtIamNwQMLhjDw6pSh3JFUwwJAF/eZbKdGGN9TU9dDVtY2x2Wc2jt37uL+ fHEsT14BHRdn/P054Fsi3uJtod4tD+ug/YeTYP7+UM91L1QNHUzOwz/YJnsh0qRJk7hkl9CBBsb3 vLzD3MvhzfMa6GgAAi9E7RXlYDy0D2z3rnUWGwmN7JnvE5MF2vnLZPNw1xVjYmKgqqpasPwCa2lC Qy/BggXzualeiemuRrv2jRnpoE+MA9st56yg67ie+uy0WBiy9UfQLVrTb4CjDRg60tLoOXvexvz8 I11pggEJOm9ob++ayI0c2Hj47MlevLfr4TzcRiYkxIuqwOovTBN4NbyIWihRqB4/gxxQvAJMGPnY 8Q8ENkT6VlDiEpKHYfioqKgQLMnAcjss2Rg0oKtDZoA2IZXUwZCvWp08NSuS0fRgYHJyMilV2SL4 RYppYMmg68iDCn0MoKy/iF1Eio6WkaKjsP734CBSnUsjIT5tLO7bExPXQWnpZZqY43Gl0oJSBQTN 4yZ2r4IuQEqisTTaJy0ddO+uJOen9JRw96TB15Ls6Uo8mnrmdNCuJp69cCFgnuZ5pQEBOuZidHFr wGe2Z0X8g/WleD28WGprgZ0gPcU6WIF2tNvroDsa839pP5cfRwP95f0HN8yA0D1xOrwAAAAASUVO RK5CYII= --f46d043893d55c785f04cbba0e10--