The commitlog_total_space_in_mb was not set, I set it to avoid having the same problem in the future.<div><br></div><div>I am aware of the over-counting problem introduced by the counters. The point is that I use them to make statistics per hours. I can understand having some wrong counts in the column corresponding to the crash time, but how to explain that all my counts since the start (months ago) have become wrong after the crash ?</div>


<div><br></div><div>After the crash I tried to repair my entire keyspace from one of the 2 nodes and this made my server crash again, no idea why. Can this failed repair be at the origin of the corrupted data ?</div><div>


<br></div><div>I&#39;m still replaying all my counts of the past months and I&#39;m afraid this kind of bug could happen again...</div><div><br></div><div>I was using cassandra for months without any issue.</div><div><br>

</div><div>Alain</div><div><br><div class="gmail_quote">2012/4/11 aaron morton <span dir="ltr">&lt;<a href="mailto:aaron@thelastpickle.com" target="_blank">aaron@thelastpickle.com</a>&gt;</span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><blockquote type="cite"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">


<div>However after recovering from this issue (freeing some space and fixing the value of  &quot;commitlog_total_space_in_mb&quot; in cassandra.yaml)</div></blockquote></div></div></blockquote></div>Did the commit log grow larger than commitlog_total_space_in_mb ? <div>


<br></div><div><div><blockquote type="cite"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">


<div>I realized that all statistics were all destroyed. I have bad values on every single counter since I start using them (september) !</div></blockquote></div></div></blockquote></div>Counter operations are not idempotent. If you client retries a counter operation it may result in the increment been applied twice. Could this have been your issue ? </div>


<div><br></div><div>Cheers</div><div><br></div><div> <br><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">


<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">


<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">


<div><div>-----------------</div><div>Aaron Morton</div><div>Freelance Developer</div><div>@aaronmorton</div><div><a href="http://www.thelastpickle.com" target="_blank">http://www.thelastpickle.com</a></div></div></div></span></div>


</span></div></span></span>
</div><div><div>

<br><div><div>On 11/04/2012, at 2:35 AM, Alain RODRIGUEZ wrote:</div><br><blockquote type="cite">By the way, I am using Cassandra 1.0.7, CL = ONE (R/W), RF = 2, 2 EC2 c1.medium nodes cluster<div><br></div><div>Alain<br><br>


<div class="gmail_quote">2012/4/10 Alain RODRIGUEZ <span dir="ltr">&lt;<a href="mailto:arodrime@gmail.com" target="_blank">arodrime@gmail.com</a>&gt;</span><br>

<blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>Hi, I&#39;m experimenting a strange and very annoying phenomena.</div>


<div><br></div><div>I had a problem with the commit log size which grew too much and full one of the hard disks in all my nodes almost at the same time (2 nodes only, RF=2, so the 2 nodes are behaving exactly in the same way)</div>





<div><br></div><div>My data are mounted in an other partition that was not full. However after recovering from this issue (freeing some space and fixing the value of  &quot;commitlog_total_space_in_mb&quot; in cassandra.yaml) I realized that all statistics were all destroyed. I have bad values on every single counter since I start using them (september) !</div>





<div><br></div><div>Does anyone experimented something similar or have any clue on this ?</div><div><br></div><div>Do you need more information ?</div><span><font color="#888888"><div><br></div><div>Alain</div>


</font></span></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div>