cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Persistently increasing read latency
Date Fri, 04 Dec 2009 01:12:56 GMT
those will be removed on GC or restart, and are harmless

On Thu, Dec 3, 2009 at 6:20 PM, B. Todd Burruss <bburruss@real.com> wrote:
> another note on this, i stopped my client and after about 35 minutes the
> compaction did complete, no more pending in compaction-pool.  however
> the Index, Data, and Filter files still exist with lots of data in them.
> "Compact" files exist for all but 4 of the Data files - these "compact"
> files are zero length.
>
> thx!
>
>
> On Thu, 2009-12-03 at 15:40 -0800, B. Todd Burruss wrote:
>> in my situation it seems like the compaction process is being starved.
>> i'm hitting the server hard for the last 45 minutes and the compaction
>> pool is sitting at 1 active, 25 pending, and 7 completed.  it has been
>> at 1 active and 7 completed for about 20 minutes.  the pending have been
>> growing steadily since then.  and as i was typing it finally finished
>> another compaction, so they must be just taking forever.
>>
>> snapshots of nodeprobe and iostats follow:
>>
>> Pool Name                    Active   Pending      Completed
>> FILEUTILS-DELETE-POOL             0         0            116
>> MESSAGING-SERVICE-POOL            0         0              0
>> STREAM-STAGE                      0         0              0
>> RESPONSE-STAGE                    0         0              0
>> ROW-READ-STAGE                    1         4        8652560
>> LB-OPERATIONS                     0         0              0
>> COMMITLOG                         1         0       14695623
>> MESSAGE-DESERIALIZER-POOL         0         0              0
>> GMFD                              0         0            
 0
>> LB-TARGET                         0         0              0
>> CONSISTENCY-MANAGER               0         0              0
>> ROW-MUTATION-STAGE                1         1       14692604
>> MESSAGE-STREAMING-POOL            0         0              0
>> LOAD-BALANCER-STAGE               0         0              0
>> FLUSH-SORTER-POOL                 0         0             28
>> MEMTABLE-POST-FLUSHER             0         0             28
>> COMPACTION-POOL                   1        25              7
>> FLUSH-WRITER-POOL                 0         0             28
>> HINTED-HANDOFF-POOL               0         0              0
>>
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>           61.85    0.00   26.68    7.73    0.00    3.74
>>
>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
>> sda             246.00     11456.00     18528.00      11456      18528
>> sda2          23074.00        20.50      1854.00         20    
  1854
>> sda1              0.00         0.00         0.00          0  
       0
>>
>>
>>
>>
>> On Thu, 2009-12-03 at 17:05 -0600, Jonathan Ellis wrote:
>> > Thanks for looking into this.  Doesn't seem like there's much
>> > low-hanging fruit to make compaction faster but I'll keep that in the
>> > back of my mind.
>> >
>> > -Jonathan
>> >
>> > On Thu, Dec 3, 2009 at 4:58 PM, Freeman, Tim <tim.freeman@hp.com> wrote:
>> > >>So this is working as designed, but the design is poor because it
>> > >>causes confusion.  If you can open a ticket for this that would be
>> > >>great.
>> > >
>> > > Done, see:
>> > >
>> > >   https://issues.apache.org/jira/browse/CASSANDRA-599
>> > >
>> > >>What does iostat -x 10 (for instance) say about the disk activity?
>> > >
>> > > rkB/s is consistently high, and wkB/s varies.  This is a typical entry
with wkB/s at the high end of its range:
>> > >
>> > >>avg-cpu:  %user   %nice    %sys %iowait   %idle
>> > >>           1.52    0.00    1.70   27.49   69.28
>> > >>
>> > >>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s
   wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>> > >>sda          3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65
13144.06   342.04    17.75   92.25   5.98  91.92
>> > >>sda1         0.00   0.00  0.00  0.00    0.00    0.00    
0.00     0.00     0.00     0.00    0.00   0.00   0.00
>> > >>sda2         3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65
13144.06   342.04    17.75   92.25   5.98  91.92
>> > >>sda3         0.00   0.00  0.00  0.00    0.00    0.00    
0.00     0.00     0.00     0.00    0.00   0.00   0.00
>> > >
>> > > and at the low end:
>> > >
>> > >>avg-cpu:  %user   %nice    %sys %iowait   %idle
>> > >>           1.50    0.00    1.77   25.80   70.93
>> > >>
>> > >>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s
   wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>> > >>sda          3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40
 3300.00   235.33     6.13   56.63   6.21  90.81
>> > >>sda1         0.00   0.00  0.00  0.00    0.00    0.00    
0.00     0.00     0.00     0.00    0.00   0.00   0.00
>> > >>sda2         3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40
 3300.00   235.33     6.13   56.63   6.21  90.81
>> > >>sda3         0.00   0.00  0.00  0.00    0.00    0.00    
0.00     0.00     0.00     0.00    0.00   0.00   0.00
>> > >
>> > > Tim Freeman
>> > > Email: tim.freeman@hp.com
>> > > Desk in Palo Alto: (650) 857-2581
>> > > Home: (408) 774-1298
>> > > Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and
Thursday; call my desk instead.)
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Jonathan Ellis [mailto:jbellis@gmail.com]
>> > > Sent: Thursday, December 03, 2009 2:45 PM
>> > > To: cassandra-user@incubator.apache.org
>> > > Subject: Re: Persistently increasing read latency
>> > >
>> > > On Thu, Dec 3, 2009 at 4:34 PM, Freeman, Tim <tim.freeman@hp.com>
wrote:
>> > >>>Can you tell if the system is i/o or cpu bound during compaction?
>> > >>
>> > >> It's I/O bound.  It's using ~9% of 1 of 4 cores as I watch it, and
all it's doing right now is compactions.
>> > >
>> > > What does iostat -x 10 (for instance) say about the disk activity?
>> > >
>>
>>
>
>
>

Mime
View raw message