Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 043E018B5F for ; Wed, 15 Jul 2015 21:04:37 +0000 (UTC) Received: (qmail 84360 invoked by uid 500); 15 Jul 2015 21:04:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84327 invoked by uid 500); 15 Jul 2015 21:04:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84317 invoked by uid 99); 15 Jul 2015 21:04:30 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2015 21:04:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F031E18019F for ; Wed, 15 Jul 2015 21:04:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.12 X-Spam-Level: *** X-Spam-Status: No, score=3.12 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_HUGEIMGSRC=0.2, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, T_REMOTE_IMAGE=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=datastax.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id z0Uyu48EfoqO for ; Wed, 15 Jul 2015 21:04:15 +0000 (UTC) Received: from mail-qk0-f181.google.com (mail-qk0-f181.google.com [209.85.220.181]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 2A2C721143 for ; Wed, 15 Jul 2015 21:04:14 +0000 (UTC) Received: by qkdl129 with SMTP id l129so37539590qkd.0 for ; Wed, 15 Jul 2015 14:04:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; s=google; h=content-type:mime-version:subject:to:from:date:message-id :in-reply-to:references; bh=TLBVrC+ajTSQ9VtMNdPYtrt4f2mRIzybiW2i78u/HW4=; b=mjq9kD9oGZzuPw+a0MjEzMc3XEfg6KVNu021CTDmxO87GGUdFJajRIY8h0VEH7rcfs VKLRbV+RhaEi9vT44Uag/6noTFMPo7kr631BxJVHyKId1xf9D0W/RaDaQObDvoOQuHEY /rj3U0aZSRBlV0NVNRvS6Fyrues3IIvXuAv60= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:to:from:date :message-id:in-reply-to:references; bh=TLBVrC+ajTSQ9VtMNdPYtrt4f2mRIzybiW2i78u/HW4=; b=krZftK2ir+WkONxxVb7SB2f4VP/490kWXurhiv11WTC5aFqGBprzUWVnseRnM338wE TwxYVypJkQl+B9ApBCpXvvXyFjLfHiota4zOXOwl0nzu8V26nE0DwSssVpoH9WBX5/fP mcBVOAqyby+zmt6ARyd9y5iYNk2vaxATyvClrY7o4KUIdaGkvuEZy35Umpckzms2CtHK dFLfNm64BW0ToS539BJkvIT15fU9o1uOIXM/IA5YEJb+jyvRZCBAutvwOoMTsetOI+ro RIwEH5rXrhovuAE/WAZ71+B8Bt0PX3O7/a7SY4pDHi/b76uiMEzsgkBtY30ra04zdZLW x0zw== X-Gm-Message-State: ALoCoQnsciDlqOuV1rbnFnPZ85TAy6gVxASrpZ8sGwj0B1vFHZGAv9Ai4nO9NDUjOF/Bk0j49mae X-Received: by 10.140.232.200 with SMTP id d191mr5197070qhc.84.1436994246129; Wed, 15 Jul 2015 14:04:06 -0700 (PDT) Received: from [127.0.0.1] (ec2-54-224-191-1.compute-1.amazonaws.com. [54.224.191.1]) by smtp.gmail.com with ESMTPSA id y11sm2894778qky.42.2015.07.15.14.04.04 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 15 Jul 2015 14:04:05 -0700 (PDT) Content-Type: multipart/mixed; boundary="===============8933921841207667224==" MIME-Version: 1.0 Subject: Re: OpsCenter datastax-agent 300% CPU To: user@cassandra.apache.org From: Sebastian Estevez Date: Wed, 15 Jul 2015 21:04:04 -0000 Message-ID: <20150715190400.16150.15958@domU-12-31-39-0A-A0-4F> In-Reply-To: References: --===============8933921841207667224== Content-Type: multipart/alternative; boundary="===============7654548026653092215==" MIME-Version: 1.0 --===============7654548026653092215== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable OpsCenter 5.2 has a couple of fixes that may result in the symptoms you = described: http://docs.datas tax.com/en/opscenter/5.2/opsc/release_notes/opscReleaseNotes520.html - Fixed issues with agent OOM when storing metrics for large numbers of = tables. (OPSC-5934 - Improved handling of metrics overflow queue on agent. (OPSC-4618) It's also got a lot of other great new features -- = http://docs.datastax.com/en/opscenter/5.2/opsc/online_help/services/opscPer= formanceService.html Let us know if this stops once you upgrade. All the best, [image: datastax_logo.png] Sebasti=C3=A1n Est=C3=A9vez Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com [image: linkedin.png] [image: = facebook.png] [image: twitter.png] = [image: g+.png] = = DataStax is the fastest, most scalable distributed database technology, = delivering Apache Cassandra to the world=E2=80=99s most innovative enterpri= ses. = Datastax is built to be agile, always-on, and predictably scalable to any = size. With more than 500 customers in 45 countries, DataStax is the = database technology and transactional backbone of choice for the worlds = most innovative companies such as Netflix, Adobe, Intuit, and eBay. = On Tue, Jul 14, 2015 at 4:40 PM, Mikhail Strebkov = wrote: > Looks like it dies with OOM: = > https://gist.github.com/kluyg/03785041e16333015c2c > > On Tue, Jul 14, 2015 at 12:01 PM, Mikhail Strebkov = > wrote: > >> OpsCenter 5.1.3 and datastax-agent-5.1.3-standalone.jar >> >> On Tue, Jul 14, 2015 at 12:00 PM, Sebastian Estevez < >> sebastian.estevez@datastax.com> wrote: >> >>> What version of the agents and what version of OpsCenter are you runnin= g? >>> >>> I recently saw something like this and upgrading to matching versions = >>> fixed the issue. >>> On Jul 14, 2015 2:58 PM, "Mikhail Strebkov" wrote: >>> >>>> Hi everyone, >>>> >>>> Recently I've noticed that most of the nodes have OpsCenter agents = >>>> running at 300% CPU. Each node has 4 cores, so agents are using 75% of = >>>> total available CPU. >>>> >>>> We're running 5 nodes with OpenSource Cassandra 2.1.8 in AWS using = >>>> Community AMI. OpsCenter version is 5.1.3. We're using Oracle Java ver= sion = >>>> 1.8.0_45. >>>> >>>> * PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND* >>>> 31501 cassandr 20 0 3599m 296m 14m S *339* 2.0 48:20.39 = >>>> /opt/jdk/jdk1.8.0_45/bin/java -Xmx128M -Djclouds.mpu.parts.magnitude= =3D100000 = >>>> -Djclouds.mpu.parts.size=3D16777216 = >>>> -Dopscenter.ssl.trustStore=3D/var/lib/datastax-agent/ssl/agentKeyStore = >>>> -Dopscenter.ssl.keyStore=3D/var/lib/datastax-agent/ssl/agentKeyStore = >>>> -Dopscenter.ssl.keyStorePassword=3Dopscenter = >>>> -Dagent-pidfile=3D/var/run/datastax-agent/datastax-agent.pid = >>>> -Dlog4j.configuration=3Dfile:/etc/datastax-agent/log4j.properties = >>>> -Djava.security.auth.login.config=3D/etc/datastax-agent/kerberos.confi= g -jar = >>>> datastax-agent-5.1.3-standalone.jar = >>>> /var/lib/datastax-agent/conf/address.yaml >>>> >>>> The logs from the agent looks strange to me: = >>>> https://gist.github.com/kluyg/21f78af7adff0a940ed3 >>>> >>>> The cluster itself seems to be fine, the load is small, nothing bad in = >>>> Cassandra system.log. >>>> >>>> Does anyone know what to tune to bring it back to normal? >>>> >>>> Thanks, >>>> Mikhail >>>> >>> >> > --===============7654548026653092215== Content-Type: text/html; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable
OpsCenter 5.2 has a couple of fixes that may result in the= symptoms you described:

  • Fixed issues with agent OOM when storing metrics for large num= bers of tables. (OPSC-5934
  • Improved handling of metrics overflow queue on a= gent. (OPSC-4618)


<= span style=3D"font-size:14px">Let us know if this stops once you upgrade.

All the best,


3D"datastax_logo.png"

Seb= asti=C3=A1n Est=C3=A9vez

Solutions Architect | 954 905 861= 5 | seb= astian.estevez@datastax.com

3D"facebook.png" 3D"twitter.png" 3D"g+.png" =



DataStax i= s the fastest, most scalable distributed database technology, delivering Apache = Cassandra to the world=E2=80=99s most innovative enterprises. Datastax is b= uilt to be agile, always-on, and predictably scalable to any size. With mor= e than 500 customers in 45 countries, DataStax is= the database technology and transactional backbone of choice for the world= s most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Tue, Jul 14, 2015 at 4:40 PM, Mikhail Str= ebkov <strebkov@gmail.com> wrote:

O= n Tue, Jul 14, 2015 at 12:01 PM, Mikhail Strebkov <strebkov@gmail.com= > wrote:
OpsCe= nter 5.1.3 and=C2=A0datastax-agent-5.1.3-standalone.jar

On Tue, Jul 14, 2015 = at 12:00 PM, Sebastian Estevez <sebastian.estevez@datastax.co= m> wrote:

Wh= at version of the agents and what version of OpsCenter are you running?

I recently saw something like this and upgrading to matching= versions fixed the issue.

On Jul 14, 2015 2:58 PM, "Mikhail Strebkov&= quot; <strebkov@= gmail.com> wrote:
Hi everyone,

Recently I've not= iced that most of the nodes have OpsCenter agents running at 300% CPU. Each= node has 4 cores, so agents are using 75% of total available CPU.

We're running 5 nodes with OpenSource Cassandra 2.1.8 = in AWS using Community AMI. OpsCenter version=C2=A0is 5.1.3. We're usin= g Oracle Java version 1.8.0_45.

=C2=A0 PID USER =C2=A0 =C2=A0 =C2=A0PR =C2=A0NI = =C2=A0VIRT =C2=A0RES =C2=A0SHR S %CPU %MEM =C2=A0 =C2=A0TIME+ =C2=A0COMMAND=
31501 cassandr = =C2=A020 =C2=A0 0 3599m 296m =C2=A014m S =C2=A0339 =C2=A02.0 =C2=A04= 8:20.39 /opt/jdk/jdk1.8.0_45/bin/java -Xmx128M -Djclouds.mpu.parts.magnitud= e=3D100000 -Djclouds.mpu.parts.size=3D16777216 -Dopscenter.ssl.trustStore= =3D/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStore=3D/va= r/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStorePassword=3Do= pscenter -Dagent-pidfile=3D/var/run/datastax-agent/datastax-agent.pid -Dlog= 4j.configuration=3Dfile:/etc/datastax-agent/log4j.properties -Djava.securit= y.auth.login.config=3D/etc/datastax-agent/kerberos.config -jar datastax-age= nt-5.1.3-standalone.jar /var/lib/datastax-agent/conf/address.yaml

The= logs from the agent looks strange to me:=C2=A0https://gist.github.co= m/kluyg/21f78af7adff0a940ed3

The cluster itsel= f seems to be fine, the load is small, nothing bad in Cassandra system.log.=

Does anyone know what to tune to bring it back to= normal?

Thanks,
Mikhail



--===============7654548026653092215==-- --===============8933921841207667224==--