incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Continuing high CPU usage (98%) after cassandra1.2.0 startup.
Date Thu, 21 Mar 2013 16:48:08 GMT
You'll need to explain a bit about how you use cassandra.

Is compaction running ? Check with nodetool compactionstats.
Are you doing very large reads? Larger than thrift_framed_transport_size_in_mb ? 

cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 20/03/2013, at 10:59 PM, 杨辉强 <huiqiangyang@yunrang.com> wrote:

> On 03/20/2013 11:34 AM, 杨辉强 wrote:
>> Hi,
>>   Every time I restart the cassandra server, the cpu usage continue to be very high(98%)
for days. But I have no reading or writing to this server.
>> I have tried the follow cmd:
>> 
>> date; date `date +"%m%d%H%M%C%y.%S"`; date;
>> 
>> It doesn't work.
>> 
>> The tail of system.log:
>> DEBUG [Thrift:1701] 2013-03-20 00:01:14,229 CustomTThreadPoolServer.java (line 209)
Thrift transport error occurred during processing of message.
>> org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken
pipe
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>> at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>> ... 7 more
>> DEBUG [Thrift:2212] 2013-03-20 00:01:14,217 CassandraServer.java (line 341) multiget_slice
>> DEBUG [Thrift:2380] 2013-03-20 00:01:14,212 CustomTThreadPoolServer.java (line 209)
Thrift transport error occurred during processing of message.
>> org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken
pipe
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>> at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>> ... 7 more
>> DEBUG [Thrift:2379] 2013-03-20 00:01:14,207 CustomTThreadPoolServer.java (line 209)
Thrift transport error occurred during processing of message.
>> org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken
pipe
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>> at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.net.SocketException: Broken pipe
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>> at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>> ... 7 more
>> 
>> 
>> If you known anything about it, please help me.
>> 
>> 
>> Thanks,
>> 
>> China.
>> 
> My log continues printing something like this:
> DEBUG [CompactionExecutor:38] 2013-03-20 17:54:35,786 IntervalTree.java (line 227) Creating
IntervalNode from [[DecoratedKey(9524159dcd3c09dd, 39353234313539646364336330396464), DecoratedKey(9526c6c7609e8afc,
39353236633663373630396538616663)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564096-Data.db')),
[DecoratedKey(9526ca7e21f4ff60, 39353236636137653231663466663630), DecoratedKey(95299a45637d63ef,
39353239396134353633376436336566)](SSTableReader(path='/mnt/storage2/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564097-Data.db'))]
> DEBUG [ReadStage:49] 2013-03-20 17:54:35,784 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:12] 2013-03-20 17:54:35,784 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:15] 2013-03-20 17:54:35,782 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:60] 2013-03-20 17:54:35,780 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:57] 2013-03-20 17:54:35,779 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 287
> DEBUG [ReadStage:45] 2013-03-20 17:54:35,778 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:9] 2013-03-20 17:54:35,775 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> INFO [ScheduledTasks:1] 2013-03-20 17:54:35,774 StatusLogger.java (line 85) MessagingService
               n/a       0,0
> DEBUG [ReadStage:38] 2013-03-20 17:54:35,774 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:55] 2013-03-20 17:54:35,773 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:18] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:32] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 287
> DEBUG [ReadStage:58] 2013-03-20 17:54:35,771 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:46] 2013-03-20 17:54:35,770 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 250
> DEBUG [ReadStage:2] 2013-03-20 17:54:35,768 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:26] 2013-03-20 17:54:35,767 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 287
> DEBUG [CompactionExecutor:36] 2013-03-20 17:54:35,767 IntervalTree.java (line 227) Creating
IntervalNode from [[DecoratedKey(9258cd03a3d015cc, 39323538636430336133643031356363), DecoratedKey(925b171b046cac97,
39323562313731623034366361633937)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-677405-Data.db'))]
> DEBUG [ReadStage:67] 2013-03-20 17:54:35,767 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:20] 2013-03-20 17:54:35,765 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 250
> DEBUG [ReadStage:36] 2013-03-20 17:54:35,763 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> DEBUG [ReadStage:65] 2013-03-20 17:54:35,762 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 287
> DEBUG [ReadStage:16] 2013-03-20 17:54:35,758 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 287
> DEBUG [ReadStage:5] 2013-03-20 17:54:35,758 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:63] 2013-03-20 17:54:35,757 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 290
> DEBUG [ReadStage:27] 2013-03-20 17:54:35,757 SSTableReader.java (line 735) Bloom filter
allows skipping sstable 288
> ......
> 
> And the cpu usage is still very high.  Is it a bug of cassandra?


Mime
View raw message