cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Jones <MJo...@imagehawk.com>
Subject RE: Some insight into the slow read speed. Where to go from here? RC1 MESSAGE-DESERIALIZER-POOL
Date Thu, 08 Apr 2010 15:46:34 GMT
I restarted Cassandra on that node to clear out that queue, reduced the available memory to
java to 4GB, and now I'm able to read with 8 concurrent threads, about 110/second

Running iostat -x I see a large amount of time in await, and a small amount of time in svctm
indicating the device is responding quickly but the queue length is excessive.

All the data filesystems are xfs on 64 bit debian, Cassandra is the only thing reading/writing
them, the CPUs are napping

Linux 2.6.31-14-generic (record)        04/08/2010      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.77    0.06    5.62   14.32    0.00   62.24

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await
 svctm  %util
sda              14.56    87.14    5.62    2.02   181.05   712.94   116.98     0.14   18.66
  4.64   3.55
sdb               0.03     0.00    0.02    0.00     0.34     0.01    14.22     0.00   10.00
  9.99   0.02
sdc               0.18    83.79   91.08    2.13  4159.40  2024.88    66.34     5.35   57.38
  3.43  31.94

$ iostat -x
Linux 2.6.31-14-generic (record)        04/08/2010      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.77    0.06    5.62   14.32    0.00   62.24

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await
 svctm  %util
sda              14.56    87.13    5.62    2.02   181.04   712.92   116.98     0.14   18.66
  4.64   3.55
sdb               0.03     0.00    0.02    0.00     0.34     0.01    14.22     0.00   10.00
  9.99   0.02
sdc               0.18    83.78   91.09    2.13  4159.65  2024.82    66.34     5.35   57.38
  3.43  31.95

-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Thursday, April 08, 2010 10:12 AM
To: user@cassandra.apache.org
Subject: Re: Some insight into the slow read speed. Where to go from here? RC1 MESSAGE-DESERIALIZER-POOL

Have you checked iostat -x ?

On Thu, Apr 8, 2010 at 9:45 AM, Mark Jones <MJones@imagehawk.com> wrote:
> I don't see any way to increase the # of active Deserializers in storage-conf.xml
>
> Tpstats more than 8 hours after insert/read stop
>
> Pool Name                    Active   Pending      Completed
> FILEUTILS-DELETE-POOL             0         0            227
> STREAM-STAGE                      0         0              1
> RESPONSE-STAGE                    0         0       76724280
> ROW-READ-STAGE                    8      4091        1138277
> LB-OPERATIONS                     0         0              0
> MESSAGE-DESERIALIZER-POOL         1   1849826       78135012
> GMFD                              0         0         136886
> LB-TARGET                         0         0              0
> CONSISTENCY-MANAGER               0         0           1803
> ROW-MUTATION-STAGE                0         0       68669717
> MESSAGE-STREAMING-POOL            0         0              0
> LOAD-BALANCER-STAGE               0         0              0
> FLUSH-SORTER-POOL                 0         0              0
> MEMTABLE-POST-FLUSHER             0         0            438
> FLUSH-WRITER-POOL                 0         0            438
> AE-SERVICE-STAGE                  0         0              3
> HINTED-HANDOFF-POOL               0         0              3
>
> More than 30 minutes later (with no reads or writes to the cluster)
>
> Pool Name                    Active   Pending      Completed
> FILEUTILS-DELETE-POOL             0         0            227
> STREAM-STAGE                      0         0              1
> RESPONSE-STAGE                    0         0       76724280
> ROW-READ-STAGE                    8      4098        1314304
> LB-OPERATIONS                     0         0              0
> MESSAGE-DESERIALIZER-POOL         1   1663578       78336771
> GMFD                              0         0         142651
> LB-TARGET                         0         0              0
> CONSISTENCY-MANAGER               0         0           1803
> ROW-MUTATION-STAGE                0         0       68669717
> MESSAGE-STREAMING-POOL            0         0              0
> LOAD-BALANCER-STAGE               0         0              0
> FLUSH-SORTER-POOL                 0         0              0
> MEMTABLE-POST-FLUSHER             0         0            438
> FLUSH-WRITER-POOL                 0         0            438
> AE-SERVICE-STAGE                  0         0              3
> HINTED-HANDOFF-POOL               0         0              3
>
> The other 2 nodes in the cluster have Pending Counts of 0, but this node seems hung
> indefinitely processing requests that should have long ago timed out for the client.
>
> TOP is showing a huge amount of I/O Wait, but I'm not sure how to track where the wait
is happening below here.  I now have jconsole up and running on this machine, and the memory
usage appears to be a saw tooth wave, going from 1GB up to 4GB over 3 hours, then plunging
back to 1GB and resuming its climb.
>
> top - 08:33:40 up 1 day, 19:25,  4 users,  load average: 7.75, 7.96, 8.16
> Tasks: 177 total,   2 running, 175 sleeping,   0 stopped,   0 zombie
> Cpu(s): 16.6%us,  7.2%sy,  0.0%ni, 34.5%id, 41.1%wa,  0.0%hi,  0.6%si,  0.0%st
> Mem:   8123068k total,  8062240k used,    60828k free,     2624k buffers
> Swap: 12699340k total,  1951504k used, 10747836k free,  3757300k cached
>

Mime
View raw message