cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Steuernol <dan...@sendwithus.com>
Subject Re: Nodes stopping
Date Thu, 11 May 2017 21:29:17 GMT
Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G
in the jvm.options for cassandra, and there is 30G on the machine.<br><br><signature></signature>
          <div class="gmail_quote nylas-quote nylas-quote-id-2ecc69e6052fbaf8640b2ea4b22e7e97d5a30384905d9f3a824012eaef30e7af">
            <br>
            On May 11 2017, at 2:22 pm, Cogumelos Maravilha &lt;cogumelosmaravilha@sapo.pt&gt;
wrote:
            <br>
            <blockquote class="gmail_quote"
              style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
              
  
    <meta content="text/html; charset=windows-1252" />
  
  <body bgcolor="#FFFFFF">
    <p>Have a look at dmesg. It have already happened to me regarding
      type i instances at AWS.</p>
    <br />
    <div>On 11-05-2017 22:17, Daniel Steuernol
      wrote:<br />
    </div>
    <blockquote type="cite">I had 2 nodes go down today, here is the ERRORs from
      the system log on both nodes
      <div>
        <div><a href="https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218">https://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218</a><br
/>
        </div>
        <div>For some context, I'm trying to get regular repairs going
          but am having issues with it.</div>
        <br />
        </div>
      <div>
        <br />
        On May 11 2017, at 2:10 pm, Cogumelos Maravilha
        <a href="mailto:cogumelosmaravilha@sapo.pt">&lt;cogumelosmaravilha@sapo.pt&gt;</a>
wrote: <br />
        <blockquote style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex;">
          <meta content="text/html; charset=windows-1252" />
          <p>Can you grep ERROR system.log<br />
          </p>
          <br />
          <div>On 11-05-2017 21:52, Daniel Steuernol wrote:<br />
          </div>
          <blockquote type="cite">There is nothing in the system log
            about it being drained or shutdown, I'm not sure how else it
            would be pre-empted. No one else on the team is on the
            servers and I haven't been shutting them down. There also is
            no java memory dump on the server either. It appears that
            the process just died.<br />
            <br />
            <div> <br />
              On May 11 2017, at 1:36 pm, Varun Gupta <a href="mailto:varung@uber.com">&lt;varung@uber.com&gt;</a>
              wrote: <br />
              <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc
                solid;padding-left:1ex;">
                <div dir="ltr"><br />
                  <div>What do you mean by &quot;no obvious error in the
                    logs&quot;, do you see node was drained or shutdown. Are
                    you sure, no other process is calling nodetool drain
                    or shutdown, OR pre-empting cassandra process?</div>
                </div>
                <div><br />
                  <div>On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol
                    <span dir="ltr">&lt;<a href="mailto:daniel@sendwithus.com"
target="_blank">daniel@sendwithus.com</a>&gt;</span>
                    wrote:<br />
                    <blockquote style="margin:0 0 0 .8ex;border-left:1px
                      #ccc solid;padding-left:1ex"><br />
                      I have a 6 node cassandra cluster running, and
                      frequently a node will go down with no obvious
                      error in the logs. This is starting to happen
                      quite often, almost daily now. Any suggestions on
                      how to track down what is causing the node to
                      stop? ------------------------------<wbr></wbr>------------------------------<wbr></wbr>---------
                      To unsubscribe, e-mail: <a href="mailto:user-unsubscribe@cassandra.apache.org"
target="_blank">user-unsubscribe@cassandra.<wbr></wbr>apache.org</a>
                      For additional commands, e-mail: <a href="mailto:user-help@cassandra.apache.org"
target="_blank">user-help@cassandra.apache.org</a>
                    </blockquote>
                  </div>
                  <br />
                </div>
              </blockquote>
            </div>
--------------------------------------------------------------------- To
            unsubscribe, e-mail: <a href="mailto:user-unsubscribe@cassandra.apache.org">user-unsubscribe@cassandra.apache.org</a>
            For additional commands, e-mail: <a href="mailto:user-help@cassandra.apache.org">user-help@cassandra.apache.org</a>
          </blockquote>
          <br />
        </blockquote>
      </div>
---------------------------------------------------------------------
      To unsubscribe, e-mail: <a href="mailto:user-unsubscribe@cassandra.apache.org">user-unsubscribe@cassandra.apache.org</a>
      For additional commands, e-mail: <a href="mailto:user-help@cassandra.apache.org">user-help@cassandra.apache.org</a>
    </blockquote>
    <br />
  </body>


            </blockquote>
          </div>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message