cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Pending on ReadStage
Date Sat, 07 Jan 2012 08:42:20 GMT
It may be IO, it's a good place to look 
http://spyced.blogspot.com/2010/01/linux-performance-basics.html

Look at the logs, when messages are dropped you will see logs from StatusLogger that include
TP stats and some other things. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 7/01/2012, at 8:35 AM, Daning Wang wrote:

> Thanks for your reply.
> 
> Nodes are equally balanced. and it is RandomPartitioner. I think that machine is slower,
Are you saying it is IO issue?
> 
> Daning
> 
> On Fri, Jan 6, 2012 at 10:25 AM, Mohit Anchlia <mohitanchlia@gmail.com> wrote:
> Are all your nodes equally balanced in terms of read requests? Are you
> using RandomPartitioner? Are you reading using indexes?
> 
> First thing you can do is compare iostat -x output between the 2 nodes
> to rule out any io issues assuming your read requests are equally
> balanced.
> 
> On Fri, Jan 6, 2012 at 10:11 AM, Daning Wang <daning@netseer.com> wrote:
> > Hi all,
> >
> > We have 5 nodes cluster(0.8.6), but the performance from one node is way
> > behind others, I checked tpstats, It always show non-zero pending ReadStage,
> > I don't see this problem on other nodes.
> >
> > What caused the problem? I/O? Memory? Cpu usage is still low. How to fix
> > this problem?
> >
> > ~/bin/nodetool -h localhost tpstats
> > Pool Name                    Active   Pending      Completed   Blocked  All
> > time blocked
> > ReadStage                        11        15          56960
> > 0                 0
> > RequestResponseStage              0         0         606695
> > 0                 0
> > MutationStage                     0         0         538634
> > 0                 0
> > ReadRepairStage                   0         0         199997
> > 0                 0
> > ReplicateOnWriteStage             0         0              0
> > 0                 0
> > GossipStage                       0         0           5734
> > 0                 0
> > AntiEntropyStage                  0         0              0
> > 0                 0
> > MigrationStage                    0         0              0
> > 0                 0
> > MemtablePostFlusher               0         0              7
> > 0                 0
> > StreamStage                       0         0              0
> > 0                 0
> > FlushWriter                       0         0              8
> > 0                 0
> > MiscStage                         0         0              0
> > 0                 0
> > FlushSorter                       0         0              0
> > 0                 0
> > InternalResponseStage             0         0              0
> > 0                 0
> > HintedHandoff                     1         4              0
> > 0                 0
> >
> > Message type           Dropped
> > RANGE_SLICE                  0
> > READ_REPAIR                  0
> > BINARY                       0
> > READ                      9082
> > MUTATION                     0
> > REQUEST_RESPONSE             0
> >
> > Thanks you in advance.
> >
> > Daning
> >
> 


Mime
View raw message