Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
  b=5ixy13GmL0WIcIKKnlS3ZzVoWxadi9yqu549oqR0QpcEUP0j7iUqQPhLZX+95BKMJjbu+I95lx7FtdJKc/qHNag5DZZEmgbJ2sIy51dKhw7f2fj5VxDBW3YPGIU1NhKEAPz1owFxcCfJsz7eXomCkETjMtRrS1SXjfXFCBBluoI=;
Message-ID: <1368547830.22785.GenericBBA@web160901.mail.bf1.yahoo.com>
Date: Tue, 14 May 2013 09:10:30 -0700 (PDT)
From: Wei Zhu <wz1975@yahoo.com>
Reply-To: Wei Zhu <wz1975@yahoo.com>
Subject: Re: (unofficial) Community Poll for Production Operators : Repair
To: user@cassandra.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="-285128509-1093262410-1368547830=:22785"

---285128509-1093262410-1368547830=:22785
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

1) 1.1.6 on 5 nodes, 24CPU, 72 RAM =0A2) local quorum (we only have one DC =
though). We do delete through TTL =0A3) yes =0A4) once a week rolling repai=
rs -pr using cron job =0A5) it definitely has negative impact on the perfor=
mance. Our data size is around 100G per node and during repair it brings in=
 additional 60G - 80G data and created about 7K compaction (We use LCS with=
 SSTable size of 10M which was a mistake we made at the beginning). It take=
s more than a day for the compaction tasks to clear and by then the next co=
mpaction starts. We had to set client side (Hector) timeout to deal with it=
 and the SLA is still under control for now. =0ABut we had to halt go live =
for another cluster due to the unanticipated "double" the space during the =
repair. =0A=0APer Dean's question to simulate the slow response, someone in=
 the IRC mentioned a trick to start Cassandra with -f and ctrl-z and it wor=
ks for our test. =0A=0A-Wei =0A----- Original Message -----=0A=0AFrom: "Dea=
n Hiller" <Dean.Hiller@nrel.gov> =0ATo: user@cassandra.apache.org =0ASent: =
Tuesday, May 14, 2013 4:48:02 AM =0ASubject: Re: (unofficial) Community Pol=
l for Production Operators : Repair =0A=0AWe had to roll out a fix in cassa=
ndra as a slow node was slowing down our clients of cassandra in 1.2.2 for =
some reason. Every time we had a slow node, we found out fast as performanc=
e degraded. We tested this in QA and had the same issue. This means a repai=
r made that node slow which made our clients slow. With this fix which I th=
ink one our team is going to try to get it back into cassandra, the slow no=
de does not affect our clients anymore. =0A=0AI am curious though, if someo=
ne else would use the "tc" program to simulate linux packet delay on a sing=
le node, does your client's response time get much slower? We simulated a 5=
00ms delay on the node to simulate the slow node=E2=80=A6.it seems the co-o=
rdinator node was incorrectly waiting for BOTH responses on CL_QUOROM inste=
ad of just one (as itself was one as well) or something like that. (I don't=
 know too much as my colleague was the one that debugged this issue) =0A=0A=
Dean =0A=0AFrom: Alain RODRIGUEZ <arodrime@gmail.com<mailto:arodrime@gmail.=
com>> =0AReply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.=
org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> =0ADate=
: Tuesday, May 14, 2013 1:42 AM =0ATo: "user@cassandra.apache.org<mailto:us=
er@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.=
apache.org>> =0ASubject: Re: (unofficial) Community Poll for Production Ope=
rators : Repair =0A=0AHi Rob, =0A=0A1) 1.2.2 on 6 to 12 EC2 m1.xlarge =0A2)=
 Quorum R&W . Almost no deletes (just some TTL) =0A3) Yes =0A4) On each nod=
e once a week (rolling repairs using crontab) =0A5) The only behavior that =
is quite odd or unexplained to me is why a repair doesn't fix a counter mis=
match between 2 nodes. I mean when I read my counters with a CL.One I have =
inconsistency (the counter value may change anytime I read it, depending, I=
 guess, on what node I read from. Reading with CL.Quorum fixes this bug, bu=
t the data is still wrong on some nodes. About performance, it's quite expe=
nsive to run a repair but doing it in a low charge period and in a rolling =
fashion works quite well and has no impact on the service. =0A=0AHope this =
will help somehow. Let me know if you need more information. =0A=0AAlain =
=0A=0A=0A=0A2013/5/10 Robert Coli <rcoli@eventbrite.com<mailto:rcoli@eventb=
rite.com>> =0AHi! =0A=0AI have been wondering how Repair is actually used b=
y operators. If =0Apeople operating Cassandra in production could answer th=
e following =0Aquestions, I would greatly appreciate it. =0A=0A1) What vers=
ion of Cassandra do you run, on what hardware? =0A2) What consistency level=
 do you write at? Do you do DELETEs? =0A3) Do you run a regularly scheduled=
 repair? =0A4) If you answered "yes" to 3, what is the frequency of the rep=
air? =0A5) What has been your subjective experience with the performance of=
 =0Arepair? (Does it work as you would expect? Does its overhead have a =0A=
significant impact on the performance of your cluster?) =0A=0AThanks! =0A=
=0A=3DRob =0A=0A=0A
---285128509-1093262410-1368547830=:22785
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><style type=3D'text/css'>p { margin: 0; }</style></head><body><=
div style=3D'font-family: arial,helvetica,sans-serif; font-size: 10pt; colo=
r: #000000'>1) 1.1.6 on 5 nodes, 24CPU, 72 RAM<br>2) local quorum (we only =
have one DC though). We do delete through TTL<br>3) yes<br>4) once a week r=
olling repairs -pr using cron job<br>5) it definitely has negative impact o=
n the performance. Our data size is around 100G per node and during repair =
it brings in additional 60G - 80G data and created about 7K compaction (We =
use LCS with SSTable size of 10M which was a mistake we made at the beginni=
ng). It takes more than a day for the compaction tasks to clear and by then=
 the next compaction starts. We had to set client side (Hector) timeout to =
deal with it and the SLA is still under control for now.<br>But we had to h=
alt go live for another cluster due to the unanticipated "double" the space=
 during the repair. <br><br>Per Dean's question to simulate the slow
 response, someone in the IRC mentioned a trick to start Cassandra with -f =
and ctrl-z and it works for our test.<br><br>-Wei<br><hr id=3D"zwchr"><div =
style=3D"color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text=
-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt=
;"><b>From: </b>"Dean Hiller" &lt;Dean.Hiller@nrel.gov&gt;<br><b>To: </b>us=
er@cassandra.apache.org<br><b>Sent: </b>Tuesday, May 14, 2013 4:48:02 AM<br=
><b>Subject: </b>Re: (unofficial) Community Poll for Production Operators :=
 Repair<br><br>We had to roll out a fix in cassandra as a slow node was slo=
wing down our clients of cassandra in 1.2.2 for some reason. &nbsp;Every ti=
me we had a slow node, we found out fast as performance degraded. &nbsp;We =
tested this in QA and had the same issue. &nbsp;This means a repair made th=
at node slow which made our clients slow. &nbsp;With this fix which I think=
 one our team is going to try to get it back into cassandra, the slow
 node does not affect our clients anymore.<br><br>I am curious though, if s=
omeone else would use the "tc" program to simulate linux packet delay on a =
single node, does your client's response time get much slower? &nbsp;We sim=
ulated a 500ms delay on the node to simulate the slow node=E2=80=A6.it seem=
s the co-ordinator node was incorrectly waiting for BOTH responses on CL_QU=
OROM instead of just one (as itself was one as well) or something like that=
. &nbsp;(I don't know too much as my colleague was the one that debugged th=
is issue)<br><br>Dean<br><br>From: Alain RODRIGUEZ &lt;arodrime@gmail.com&l=
t;mailto:arodrime@gmail.com&gt;&gt;<br>Reply-To: "user@cassandra.apache.org=
&lt;mailto:user@cassandra.apache.org&gt;" &lt;user@cassandra.apache.org&lt;=
mailto:user@cassandra.apache.org&gt;&gt;<br>Date: Tuesday, May 14, 2013 1:4=
2 AM<br>To: "user@cassandra.apache.org&lt;mailto:user@cassandra.apache.org&=
gt;"
 &lt;user@cassandra.apache.org&lt;mailto:user@cassandra.apache.org&gt;&gt;<=
br>Subject: Re: (unofficial) Community Poll for Production Operators : Repa=
ir<br><br>Hi Rob,<br><br>1) 1.2.2 on 6 to 12 EC2 m1.xlarge<br>2) Quorum R&a=
mp;W . Almost no deletes (just some TTL)<br>3) Yes<br>4) On each node once =
a week (rolling repairs using crontab)<br>5) The only behavior that is quit=
e odd or unexplained to me is why a repair doesn't fix a counter mismatch b=
etween 2 nodes. I mean when I read my counters with a CL.One I have inconsi=
stency (the counter value may change anytime I read it, depending, I guess,=
 on what node I read from. Reading with CL.Quorum fixes this bug, but the d=
ata is still wrong on some nodes. About performance, it's quite expensive t=
o run a repair but doing it in a low charge period and in a rolling fashion=
 works quite well and has no impact on the service.<br><br>Hope this will h=
elp somehow. Let me know if you need more
 information.<br><br>Alain<br><br><br><br>2013/5/10 Robert Coli &lt;rcoli@e=
ventbrite.com&lt;mailto:rcoli@eventbrite.com&gt;&gt;<br>Hi!<br><br>I have b=
een wondering how Repair is actually used by operators. If<br>people operat=
ing Cassandra in production could answer the following<br>questions, I woul=
d greatly appreciate it.<br><br>1) What version of Cassandra do you run, on=
 what hardware?<br>2) What consistency level do you write at? Do you do DEL=
ETEs?<br>3) Do you run a regularly scheduled repair?<br>4) If you answered =
"yes" to 3, what is the frequency of the repair?<br>5) What has been your s=
ubjective experience with the performance of<br>repair? (Does it work as yo=
u would expect? Does its overhead have a<br>significant impact on the perfo=
rmance of your cluster?)<br><br>Thanks!<br><br>=3DRob<br><br></div><br></di=
v></body></html>
---285128509-1093262410-1368547830=:22785--