incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 杨辉强 <huiqiangy...@yunrang.com>
Subject Re: Continuing high CPU usage (98%) after cassandra1.2.0 startup.
Date Wed, 20 Mar 2013 09:59:53 GMT
On 03/20/2013 11:34 AM, 杨辉强 wrote:
> Hi,
>    Every time I restart the cassandra server, the cpu usage continue 
> to be very high(98%) for days. But I have no reading or writing to 
> this server.
> I have tried the follow cmd:
>
> date; date `date +"%m%d%H%M%C%y.%S"`; date;
>
> It doesn't work.
>
> The tail of system.log:
> DEBUG [Thrift:1701] 2013-03-20 00:01:14,229 
> CustomTThreadPoolServer.java (line 209) Thrift transport error 
> occurred during processing of message.
> org.apache.thrift.transport.TTransportException: 
> java.net.SocketException: Broken pipe
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> at 
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ... 7 more
> DEBUG [Thrift:2212] 2013-03-20 00:01:14,217 CassandraServer.java (line 
> 341) multiget_slice
> DEBUG [Thrift:2380] 2013-03-20 00:01:14,212 
> CustomTThreadPoolServer.java (line 209) Thrift transport error 
> occurred during processing of message.
> org.apache.thrift.transport.TTransportException: 
> java.net.SocketException: Broken pipe
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> at 
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ... 7 more
> DEBUG [Thrift:2379] 2013-03-20 00:01:14,207 
> CustomTThreadPoolServer.java (line 209) Thrift transport error 
> occurred during processing of message.
> org.apache.thrift.transport.TTransportException: 
> java.net.SocketException: Broken pipe
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> at 
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:36)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at 
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
> ... 7 more
>
>
> If you known anything about it, please help me.
>
>
> Thanks,
>
> China.
>
My log continues printing something like this:
DEBUG [CompactionExecutor:38] 2013-03-20 17:54:35,786 IntervalTree.java 
(line 227) Creating IntervalNode from [[DecoratedKey(9524159dcd3c09dd, 
39353234313539646364336330396464), DecoratedKey(9526c6c7609e8afc, 
39353236633663373630396538616663)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564096-Data.db')),

[DecoratedKey(9526ca7e21f4ff60, 39353236636137653231663466663630), 
DecoratedKey(95299a45637d63ef, 
39353239396134353633376436336566)](SSTableReader(path='/mnt/storage2/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-564097-Data.db'))]
DEBUG [ReadStage:49] 2013-03-20 17:54:35,784 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:12] 2013-03-20 17:54:35,784 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:15] 2013-03-20 17:54:35,782 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:60] 2013-03-20 17:54:35,780 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:57] 2013-03-20 17:54:35,779 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:45] 2013-03-20 17:54:35,778 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:9] 2013-03-20 17:54:35,775 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
  INFO [ScheduledTasks:1] 2013-03-20 17:54:35,774 StatusLogger.java 
(line 85) MessagingService                n/a       0,0
DEBUG [ReadStage:38] 2013-03-20 17:54:35,774 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:55] 2013-03-20 17:54:35,773 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:18] 2013-03-20 17:54:35,771 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:32] 2013-03-20 17:54:35,771 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:58] 2013-03-20 17:54:35,771 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:46] 2013-03-20 17:54:35,770 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 250
DEBUG [ReadStage:2] 2013-03-20 17:54:35,768 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:26] 2013-03-20 17:54:35,767 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 287
DEBUG [CompactionExecutor:36] 2013-03-20 17:54:35,767 IntervalTree.java 
(line 227) Creating IntervalNode from [[DecoratedKey(9258cd03a3d015cc, 
39323538636430336133643031356363), DecoratedKey(925b171b046cac97, 
39323562313731623034366361633937)](SSTableReader(path='/mnt/storage0/cassandra/WebSearch/WebDepotCF/WebSearch-WebDepotCF-ia-677405-Data.db'))]
DEBUG [ReadStage:67] 2013-03-20 17:54:35,767 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:20] 2013-03-20 17:54:35,765 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 250
DEBUG [ReadStage:36] 2013-03-20 17:54:35,763 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
DEBUG [ReadStage:65] 2013-03-20 17:54:35,762 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:16] 2013-03-20 17:54:35,758 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 287
DEBUG [ReadStage:5] 2013-03-20 17:54:35,758 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:63] 2013-03-20 17:54:35,757 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 290
DEBUG [ReadStage:27] 2013-03-20 17:54:35,757 SSTableReader.java (line 
735) Bloom filter allows skipping sstable 288
......

And the cpu usage is still very high.  Is it a bug of cassandra?

Mime
View raw message