cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Wong <gokoproj...@gmail.com>
Subject Re: OpsCenter datastax-agent 300% CPU
Date Mon, 20 Jul 2015 13:36:59 GMT
Hi all & Sebastain

We recently encountered similar issue. At least we observed agent
constantly died with OOM. Unfortunately, we are still with 1.2.X and it
will be a while before we can totally move to Cassandra 2 series.

Is there a backport patch to fix OOM in OpsCenter 5.1 branch? Please let us
know because losing OpsCenter is a huge deal for administrator.

Thank you.



On Wed, Jul 15, 2015 at 6:28 PM, Mikhail Strebkov <strebkov@gmail.com>
wrote:

> Thanks, I think it got resolved after an update.
>
> Kind regards,
> Mikhail
>
> On Wed, Jul 15, 2015 at 2:04 PM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> OpsCenter 5.2 has a couple of fixes that may result in the symptoms you
>> described:
>> http://docs.datas
>> tax.com/en/opscenter/5.2/opsc/release_notes/opscReleaseNotes520.html
>>
>>
>>    - Fixed issues with agent OOM when storing metrics for large numbers
>>    of tables. (OPSC-5934
>>    - Improved handling of metrics overflow queue on agent. (OPSC-4618)
>>
>>
>> It's also got a lot of other great new features --
>> http://docs.datastax.com/en/opscenter/5.2/opsc/online_help/services/opscPerformanceService.html
>>
>> Let us know if this stops once you upgrade.
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>>
>> <http://cassandrasummit-datastax.com/>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Tue, Jul 14, 2015 at 4:40 PM, Mikhail Strebkov <strebkov@gmail.com>
>> wrote:
>>
>>> Looks like it dies with OOM:
>>> https://gist.github.com/kluyg/03785041e16333015c2c
>>>
>>> On Tue, Jul 14, 2015 at 12:01 PM, Mikhail Strebkov <strebkov@gmail.com>
>>> wrote:
>>>
>>>> OpsCenter 5.1.3 and datastax-agent-5.1.3-standalone.jar
>>>>
>>>> On Tue, Jul 14, 2015 at 12:00 PM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> What version of the agents and what version of OpsCenter are you
>>>>> running?
>>>>>
>>>>> I recently saw something like this and upgrading to matching versions
>>>>> fixed the issue.
>>>>> On Jul 14, 2015 2:58 PM, "Mikhail Strebkov" <strebkov@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Recently I've noticed that most of the nodes have OpsCenter agents
>>>>>> running at 300% CPU. Each node has 4 cores, so agents are using 75%
of
>>>>>> total available CPU.
>>>>>>
>>>>>> We're running 5 nodes with OpenSource Cassandra 2.1.8 in AWS using
>>>>>> Community AMI. OpsCenter version is 5.1.3. We're using Oracle Java
version
>>>>>> 1.8.0_45.
>>>>>>
>>>>>> *  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND*
>>>>>> 31501 cassandr  20   0 3599m 296m  14m S  *339*  2.0  48:20.39
>>>>>> /opt/jdk/jdk1.8.0_45/bin/java -Xmx128M -Djclouds.mpu.parts.magnitude=100000
>>>>>> -Djclouds.mpu.parts.size=16777216
>>>>>> -Dopscenter.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore
>>>>>> -Dopscenter.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore
>>>>>> -Dopscenter.ssl.keyStorePassword=opscenter
>>>>>> -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid
>>>>>> -Dlog4j.configuration=file:/etc/datastax-agent/log4j.properties
>>>>>> -Djava.security.auth.login.config=/etc/datastax-agent/kerberos.config
-jar
>>>>>> datastax-agent-5.1.3-standalone.jar
>>>>>> /var/lib/datastax-agent/conf/address.yaml
>>>>>>
>>>>>> The logs from the agent looks strange to me:
>>>>>> https://gist.github.com/kluyg/21f78af7adff0a940ed3
>>>>>>
>>>>>> The cluster itself seems to be fine, the load is small, nothing bad
>>>>>> in Cassandra system.log.
>>>>>>
>>>>>> Does anyone know what to tune to bring it back to normal?
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message