cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Steuernol <dan...@sendwithus.com>
Subject Re: Restarting nodes and reported load
Date Tue, 30 May 2017 20:02:20 GMT
incremental backup is set to false in the config file, also I have set &nbsp;snapshot_before_compaction
and&nbsp;auto_snapshot to false as well. I ran nodetool clearsnapshot, but before doing
that I ran nodetool listsnapshots and it listed a bunch of snapshots. I would have expected
that to be empty because I've disabled auto_snapshot.<br><br><signature></signature>
          <div class="gmail_quote nylas-quote nylas-quote-id-2840649ff146b5a66bb14a35cc597fd94c663047db88da522cdf0304fd60ec02">
            <br>
            On May 30 2017, at 12:15 pm, Varun Gupta &lt;varung@uber.com&gt; wrote:
            <br>
            <blockquote class="gmail_quote"
              style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
              <div dir="ltr">Can you please check if you have incremental backup enabled
and snapshots are occupying the space.<div><br /></div><div>run nodetool
clearsnapshot command.</div></div><div><br /><div>On Tue, May
30, 2017 at 11:12 AM, Daniel Steuernol <span dir="ltr">&lt;<a href="mailto:daniel@sendwithus.com"
target="_blank">daniel@sendwithus.com</a>&gt;</span> wrote:<br /><blockquote
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It's 3-4TB per node,
and by load rises, I'm talking about load as reported by nodetool status.<div><div><br
/><br /><u></u><u></u>
          <div>
            <br />
            On May 30 2017, at 10:25 am, daemeon reiydelle &lt;<a href="mailto:daemeonr@gmail.com"
target="_blank">daemeonr@gmail.com</a>&gt; wrote:
            <br />
            <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr"><div style="font-family:comic sans ms,sans-serif;color:rgb(7,55,99)">When
you say &quot;the load rises ... &quot;, could you clarify what you mean by &quot;load&quot;?
That has a specific Linux term, and in e.g. Cloudera Manager. But in neither case would that
be relevant to transient or persisted disk. Am I missing something?<br /></div><div><br
/>
<br /><div>On Tue, May 30, 2017 at 10:18 AM, tommaso barbugli <span dir="ltr">&lt;<a
href="mailto:tbarbugli@gmail.com" target="_blank">tbarbugli@gmail.com</a>&gt;</span>
wrote:<br /><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir="ltr">3-4 TB per node or in total?</div><div><br /><div>On
Tue, May 30, 2017 at 6:48 PM, Daniel Steuernol <span dir="ltr">&lt;<a href="mailto:daniel@sendwithus.com"
target="_blank">daniel@sendwithus.com</a>&gt;</span> wrote:<br /><blockquote
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I should also mention
that I am running cassandra 3.10 on the cluster<div><div><br /><br /><u></u><u></u>
          <div>
            <br />
            On May 29 2017, at 9:43 am, Daniel Steuernol &lt;<a href="mailto:daniel@sendwithus.com"
target="_blank">daniel@sendwithus.com</a>&gt; wrote:
            <br />
            <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              The cluster is running with RF=3, right now each node is storing about 3-4 TB
of data. I'm using r4.2xlarge EC2 instances, these have 8 vCPU's, 61 GB of RAM, and the disks
attached for the data drive are gp2 ssd ebs volumes with 10k iops. I guess this brings up
the question of what's a good marker to decide on whether to increase disk space vs provisioning
a new node?<br /><br />
          <div>
            <br />
            On May 29 2017, at 9:35 am, tommaso barbugli &lt;<a href="mailto:tbarbugli@gmail.com"
target="_blank">tbarbugli@gmail.com</a>&gt; wrote:
            <br />
            <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">Hi Daniel,<div><br /></div><div>This
is not normal. Possibly a capacity problem. Whats the RF, how much data do you store per node
and what kind of servers do you use (core count, RAM, disk, ...)?</div><div><br
/></div><div>Cheers,</div><div>Tommaso</div></div><div><br
/><div>On Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol <span dir="ltr">&lt;<a
href="mailto:daniel@sendwithus.com" target="_blank">daniel@sendwithus.com</a>&gt;</span>
wrote:<br /><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br
/>I am running a 6 node cluster, and I have noticed that the reported load on each node
rises throughout the week and grows way past the actual disk space used and available on each
node. Also eventually latency for operations suffers and the nodes have to be restarted. A
couple questions on this, is this normal? Also does cassandra need to be restarted every few
days for best performance? Any insight on this behaviour would be helpful.<u></u><u></u><div><br
/></div><div>Cheers,</div><div>Daniel</div>

------------------------------<wbr></wbr>------------------------------<wbr></wbr>---------
To unsubscribe, e-mail: <a href="mailto:user-unsubscribe@cassandra.apache.org" target="_blank">user-unsubscribe@cassandra.apa<wbr></wbr>che.org</a>
For additional commands, e-mail: <a href="mailto:user-help@cassandra.apache.org" target="_blank">user-help@cassandra.apache.org</a>

</blockquote></div><br /></div>

            </blockquote>
          </div>
            </blockquote>
          </div>
</div></div></blockquote></div><br /></div>
</blockquote></div><br /></div></div>

            </blockquote>
          </div>

------------------------------<wbr></wbr>------------------------------<wbr></wbr>---------
To unsubscribe, e-mail: <a href="mailto:user-unsubscribe@cassandra.apache.org" target="_blank">user-unsubscribe@cassandra.<wbr></wbr>apache.org</a>
For additional commands, e-mail: <a href="mailto:user-help@cassandra.apache.org" target="_blank">user-help@cassandra.apache.org</a>

</div></div></blockquote></div><br /></div>

            </blockquote>
          </div>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message