incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Li <ale...@gmail.com>
Subject Periodically hiccups
Date Thu, 22 Apr 2010 07:40:06 GMT
Hello,

We recently deployed a cluster of 5 Cassandra nodes into production, and 
ran into big problems with periodically hiccups (individual node goes 
down, high CPU, client connection timeout). It was terrible with 0.5 
(one hiccups every 5-10 minutes), today we upgraded to 0.6.1, it happens 
less frequently now (likely once every 30 minutes or so). But it is 
still quite frustrating.

We used ReplicationFactor=3 for all column families. 5 nodes are behind 
haproxy. Java client goes through haproxy. The most obvious behavior is: 
as soon as one node goes down, the connections between haproxy and 
Cassandra nodes just shoot up to 1000 (in normal case it is stable at 
40), and the connections don't go down for quite a while. Meanwhile Java 
clients just get all kind of TimeoutException, then kept on retrying. 
Eventually we have to restart haproxy, then things go back to normal.

Each node has 5GB max heap, powerful enough CPU (quad-core), software 
RAID mirror. We are definitely NOT putting lots of load yet, mostly 
20-50 concurrent requests to Cassandra, but it is not holding up! Please 
help, we are on the verge of giving up Cassandra after 5 days of 
periodic "outage".

Couple observations:

- cfstats shows significant read latency on "system" keyspace, almost 5s 
(see below)

- RecentReadLatencyMicros and RecentWriteLatencyMicros are super high 
for StorageProxy, as well as every column family in JMX: up to 43 s and 
9s (see screenshot). However, in cfstats, they are quite small.

- Every second we see 5-10 DigestMismatchException in the log:

 INFO [pool-1-thread-15857] 2010-04-22 00:37:37,887 StorageProxy.java 
(line 499) DigestMismatchException: Mismatch for key 1068022523 
(d41d8cd98f00b204e9800998ecf8427e vs 0dd4cdaeeb1a334ae133c6955e109629)

Please advice. Thank you!

storage-conf, cfstats, tpstats are listed below:

root@cdb-006:/glass/sfw/cassandra# cat conf/storage-conf.xml
<!--
 ~ Licensed to the Apache Software Foundation (ASF) under one
 ~ or more contributor license agreements.  See the NOTICE file
 ~ distributed with this work for additional information
 ~ regarding copyright ownership.  The ASF licenses this file
 ~ to you under the Apache License, Version 2.0 (the
 ~ "License"); you may not use this file except in compliance
 ~ with the License.  You may obtain a copy of the License at
 ~
 ~    http://www.apache.org/licenses/LICENSE-2.0
 ~
 ~ Unless required by applicable law or agreed to in writing,
 ~ software distributed under the License is distributed on an
 ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 ~ KIND, either express or implied.  See the License for the
 ~ specific language governing permissions and limitations
 ~ under the License.
-->
<Storage>
    
<!--======================================================================-->
    <!-- Basic 
Configuration                                                  -->
    
<!--======================================================================-->

    <!--
       ~ The name of this cluster.  This is mainly used to prevent 
machines in
       ~ one logical cluster from joining another.
      -->
    <ClusterName>Test Cluster</ClusterName>

    <!--
       ~ Turn on to make new [non-seed] nodes automatically migrate the 
right data
       ~ to themselves.  (If no InitialToken is specified, they will 
pick one
       ~ such that they will get half the range of the most-loaded node.)
       ~ If a node starts up without bootstrapping, it will mark itself 
bootstrapped
       ~ so that you can't subsequently accidently bootstrap a node with
       ~ data on it.  (You can reset this by wiping your data and commitlog
       ~ directories.)
       ~
       ~ Off by default so that new clusters and upgraders from 0.4 don't
       ~ bootstrap immediately.  You should turn this on when you start 
adding
       ~ new nodes to a cluster that already has data on it.  (If you 
are upgrading
       ~ from 0.4, start your cluster with it off once before changing 
it to true.
       ~ Otherwise, no data will be lost but you will incur a lot of 
unnecessary
       ~ I/O before your cluster starts up.)
      -->
    <AutoBootstrap>false</AutoBootstrap>

    <!--
       ~ Keyspaces and ColumnFamilies:
       ~ A ColumnFamily is the Cassandra concept closest to a relational
       ~ table.  Keyspaces are separate groups of ColumnFamilies.  Except in
       ~ very unusual circumstances you will have one Keyspace per 
application.

       ~ There is an implicit keyspace named 'system' for Cassandra 
internals.
      -->
    <Keyspaces>
        <Keyspace Name="Pandora">
            <ColumnFamily CompareWith="BytesType" Name="Standard1"/>
            <ColumnFamily CompareWith="UTF8Type" Name="Standard2"/>
            <ColumnFamily CompareWith="TimeUUIDType" 
Name="StandardByUUID1"/>
            <ColumnFamily CompareWith="TimeUUIDType" Name="Folder1"/>
            <ColumnFamily CompareWith="UTF8Type" Name="FolderInfo"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="User"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="Message"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="Folder"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="Attachment"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>

            
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
            <ReplicationFactor>3</ReplicationFactor>
            
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
        </Keyspace>

        <Keyspace Name="Titan">
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="Club"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="Payment"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="User"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily ColumnType="Super"
                          CompareWith="UTF8Type"
                          CompareSubcolumnsWith="UTF8Type"
                          Name="FbUser"
                          Comment="A column family with supercolumns, 
whose column and subcolumn names are UTF8 strings"/>
            <ColumnFamily CompareWith="UTF8Type" Name="RandomUsers"/>

            
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
            <ReplicationFactor>3</ReplicationFactor>
            
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
        </Keyspace>
       
    </Keyspaces>

    <!--
               ~ Authenticator: any IAuthenticator may be used, 
including your own as long
               ~ as it is on the classpath.  Out of the box, Cassandra 
provides
               ~ org.apache.cassandra.auth.AllowAllAuthenticator and,
               ~ org.apache.cassandra.auth.SimpleAuthenticator
               ~ (SimpleAuthenticator uses access.properties and 
passwd.properties by
               ~ default).
               ~
               ~ If you don't specify an authenticator, 
AllowAllAuthenticator is used.
              -->
    
<Authenticator>org.apache.cassandra.auth.AllowAllAuthenticator</Authenticator>

    <!--
               ~ Partitioner: any IPartitioner may be used, including 
your own as long
               ~ as it is on the classpath.  Out of the box, Cassandra 
provides
               ~ org.apache.cassandra.dht.RandomPartitioner,
               ~ org.apache.cassandra.dht.OrderPreservingPartitioner, and
               ~ 
org.apache.cassandra.dht.CollatingOrderPreservingPartitioner.
               ~ (CollatingOPP colates according to EN,US rules, not 
naive byte
               ~ ordering.  Use this as an example if you need 
locale-aware collation.)
               ~ Range queries require using an order-preserving 
partitioner.
               ~
               ~ Achtung!  Changing this parameter requires wiping your data
               ~ directories, since the partitioner can modify the 
sstable on-disk
               ~ format.
              -->
    <Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>

    <!--
               ~ If you are using an order-preserving partitioner and 
you know your key
               ~ distribution, you can specify the token for this node 
to use. (Keys
               ~ are sent to the node with the "closest" token, so 
distributing your
               ~ tokens equally along the key distribution space will 
spread keys
               ~ evenly across your cluster.)  This setting is only 
checked the first
               ~ time a node is started.

               ~ This can also be useful with RandomPartitioner to force 
equal spacing
               ~ of tokens around the hash space, especially for 
clusters with a small
               ~ number of nodes.
              -->
    <InitialToken></InitialToken>

    <!--
               ~ Directories: Specify where Cassandra should store 
different data on
               ~ disk.  Keep the data disks and the CommitLog disks 
separate for best
               ~ performance
              -->
    <CommitLogDirectory>/var/lib/cassandra/commitlog</CommitLogDirectory>
    <DataFileDirectories>
        <DataFileDirectory>/var/lib/cassandra/data</DataFileDirectory>
    </DataFileDirectories>


    <!--
               ~ Addresses of hosts that are deemed contact points. 
Cassandra nodes
               ~ use this list of hosts to find each other and learn the 
topology of
               ~ the ring. You must change this if you are running 
multiple nodes!
              -->
    <Seeds>
        <Seed>10.10.104.11</Seed>
        <Seed>10.10.104.13</Seed>
        <Seed>10.10.104.15</Seed>
    </Seeds>


    <!-- Miscellaneous -->

    <!-- Time to wait for a reply from other nodes before failing the 
command -->
    <RpcTimeoutInMillis>10000</RpcTimeoutInMillis>
    <!-- Size to allow commitlog to grow to before creating a new 
segment -->
    <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>


    <!-- Local hosts and ports -->

    <!--
               ~ Address to bind to and tell other nodes to connect to.  
You _must_
               ~ change this if you want multiple nodes to be able to 
communicate!
               ~
               ~ Leaving it blank leaves it up to 
InetAddress.getLocalHost(). This
               ~ will always do the Right Thing *if* the node is 
properly configured
               ~ (hostname, name resolution, etc), and the Right Thing 
is to use the
               ~ address associated with the hostname (it might not be).
              -->
    <ListenAddress></ListenAddress>
    <!-- internal communications port -->
    <StoragePort>7000</StoragePort>

    <!--
               ~ The address to bind the Thrift RPC service to. Unlike 
ListenAddress
               ~ above, you *can* specify 0.0.0.0 here if you want 
Thrift to listen on
               ~ all interfaces.
               ~
               ~ Leaving this blank has the same effect it does for 
ListenAddress,
               ~ (i.e. it will be based on the configured hostname of 
the node).
              -->
    <ThriftAddress>0.0.0.0</ThriftAddress>
    <!-- Thrift RPC port (the port clients connect to). -->
    <ThriftPort>9160</ThriftPort>
    <!--
               ~ Whether or not to use a framed transport for Thrift. If 
this option
               ~ is set to true then you must also use a framed 
transport on the
               ~ client-side, (framed and non-framed transports are not 
compatible).
              -->
    <ThriftFramedTransport>false</ThriftFramedTransport>


    
<!--======================================================================-->
    <!-- Memory, Disk, and 
Performance                                        -->
    
<!--======================================================================-->

    <!--
               ~ Access mode.  mmapped i/o is substantially faster, but 
only practical on
               ~ a 64bit machine (which notably does not include EC2 
"small" instances)
               ~ or relatively small datasets.  "auto", the safe choice, 
will enable
               ~ mmapping on a 64bit JVM.  Other values are "mmap", 
"mmap_index_only"
               ~ (which may allow you to get part of the benefits of 
mmap on a 32bit
               ~ machine by mmapping only index files) and "standard".
               ~ (The buffer size settings that follow only apply to 
standard,
               ~ non-mmapped i/o.)
               -->
    <DiskAccessMode>auto</DiskAccessMode>

    <!--
               ~ Size of compacted row above which to log a warning.  
(If compacted
               ~ rows do not fit in memory, Cassandra will crash.  This 
is explained
               ~ in 
http://wiki.apache.org/cassandra/CassandraLimitations and is
               ~ scheduled to be fixed in 0.7.)
              -->
    <RowWarningThresholdInMB>512</RowWarningThresholdInMB>

    <!--
               ~ Buffer size to use when performing contiguous column 
slices. Increase
               ~ this to the size of the column slices you typically 
perform.
               ~ (Name-based queries are performed with a buffer size of
               ~ ColumnIndexSizeInKB.)
              -->
    <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>

    <!--
               ~ Buffer size to use when flushing memtables to disk. 
(Only one
               ~ memtable is ever flushed at a time.) Increase 
(decrease) the index
               ~ buffer size relative to the data buffer if you have few 
(many)
               ~ columns per key.  Bigger is only better _if_ your 
memtables get large
               ~ enough to use the space. (Check in your data directory 
after your
               ~ app has been running long enough.) -->
    <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
    <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>

    <!--
               ~ Add column indexes to a row after its contents reach 
this size.
               ~ Increase if your column values are large, or if you 
have a very large
               ~ number of columns.  The competing causes are, Cassandra 
has to
               ~ deserialize this much of the row to read a single 
column, so you want
               ~ it to be small - at least if you do many partial-row 
reads - but all
               ~ the index data is read for each access, so you don't 
want to generate
               ~ that wastefully either.
              -->
    <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>

    <!--
               ~ Flush memtable after this much data has been inserted, 
including
               ~ overwritten data.  There is one memtable per column 
family, and
               ~ this threshold is based solely on the amount of data 
stored, not
               ~ actual heap memory usage (there is some overhead in 
indexing the
               ~ columns).
              -->
    <MemtableThroughputInMB>64</MemtableThroughputInMB>
    <!--
               ~ Throughput setting for Binary Memtables.  Typically 
these are
               ~ used for bulk load so you want them to be larger.
              -->
    <BinaryMemtableThroughputInMB>256</BinaryMemtableThroughputInMB>
    <!--
               ~ The maximum number of columns in millions to store in 
memory per
               ~ ColumnFamily before flushing to disk.  This is also a 
per-memtable
               ~ setting.  Use with MemtableThroughputInMB to tune 
memory usage.
              -->
    <MemtableOperationsInMillions>0.3</MemtableOperationsInMillions>
    <!--
               ~ The maximum time to leave a dirty memtable unflushed.
               ~ (While any affected columnfamilies have unflushed data 
from a
               ~ commit log segment, that segment cannot be deleted.)
               ~ This needs to be large enough that it won't cause a 
flush storm
               ~ of all your memtables flushing at once because none has hit
               ~ the size or count thresholds yet.  For production, a larger
               ~ value such as 1440 is recommended.
              -->
    <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>

    <!--
               ~ Unlike most systems, in Cassandra writes are faster 
than reads, so
               ~ you can afford more of those in parallel.  A good rule 
of thumb is 2
               ~ concurrent reads per processor core.  Increase 
ConcurrentWrites to
               ~ the number of clients writing at once if you enable 
CommitLogSync +
               ~ CommitLogSyncDelay. -->
    <ConcurrentReads>8</ConcurrentReads>
    <ConcurrentWrites>32</ConcurrentWrites>

    <!--
               ~ CommitLogSync may be either "periodic" or "batch."  
When in batch
               ~ mode, Cassandra won't ack writes until the commit log 
has been
               ~ fsynced to disk.  It will wait up to 
CommitLogSyncBatchWindowInMS
               ~ milliseconds for other writes, before performing the sync.

               ~ This is less necessary in Cassandra than in traditional 
databases
               ~ since replication reduces the odds of losing data from 
a failure
               ~ after writing the log entry but before it actually 
reaches the disk.
               ~ So the other option is "periodic," where writes may be 
acked immediately
               ~ and the CommitLog is simply synced every 
CommitLogSyncPeriodInMS
               ~ milliseconds.
              -->
    <CommitLogSync>periodic</CommitLogSync>
    <!--
               ~ Interval at which to perform syncs of the CommitLog in 
periodic mode.
               ~ Usually the default of 10000ms is fine; increase it if 
your i/o
               ~ load is such that syncs are taking excessively long times.
              -->
    <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
    <!--
               ~ Delay (in milliseconds) during which additional commit 
log entries
               ~ may be written before fsync in batch mode.  This will 
increase
               ~ latency slightly, but can vastly improve throughput 
where there are
               ~ many writers.  Set to zero to disable (each entry will 
be synced
               ~ individually).  Reasonable values range from a minimal 
0.1 to 10 or
               ~ even more if throughput matters more than latency.
              -->
    <!-- <CommitLogSyncBatchWindowInMS>1</CommitLogSyncBatchWindowInMS> -->

    <!--
               ~ Time to wait before garbage-collection deletion 
markers.  Set this to
               ~ a large enough value that you are confident that the 
deletion marker
               ~ will be propagated to all replicas by the time this 
many seconds has
               ~ elapsed, even in the face of hardware failures.  The 
default value is
               ~ ten days.
              -->
    <GCGraceSeconds>864000</GCGraceSeconds>
</Storage>
root@cdb-006:/glass/sfw/cassandra# bin/nodetool -h localhost cfstats
Keyspace: system
    Read Count: 878
    Read Latency: 5752.042634396355 ms.
    Write Count: 2260398
    Write Latency: 0.014567047926957996 ms.
    Pending Tasks: 0
        Column Family: LocationInfo
        SSTable count: 2
        Space used (live): 3569
        Space used (total): 3569
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 1
        Read Count: 1
        Read Latency: NaN ms.
        Write Count: 6
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 2
        Key cache size: 1
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: HintsColumnFamily
        SSTable count: 2
        Space used (live): 70272035
        Space used (total): 70272035
        Memtable Columns Count: 56264
        Memtable Data Size: 486854
        Memtable Switch Count: 21
        Read Count: 877
        Read Latency: 13614.412 ms.
        Write Count: 2260392
        Write Latency: 0.142 ms.
        Pending Tasks: 0
        Key cache capacity: 2
        Key cache size: 2
        Key cache hit rate: 0.25
        Row cache: disabled
        Compacted row minimum size: 78567
        Compacted row maximum size: 39561901
        Compacted row mean size: 27878603

----------------
Keyspace: Titan
    Read Count: 8948702
    Read Latency: 7.949136100185256 ms.
    Write Count: 3393490
    Write Latency: 0.19255415398306758 ms.
    Pending Tasks: 0
        Column Family: FbUser
        SSTable count: 6
        Space used (live): 3675014807
        Space used (total): 3675014807
        Memtable Columns Count: 250055
        Memtable Data Size: 9146339
        Memtable Switch Count: 9
        Read Count: 6591406
        Read Latency: 8.361 ms.
        Write Count: 343030
        Write Latency: 0.078 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 200000
        Key cache hit rate: 0.6341912478864628
        Row cache: disabled
        Compacted row minimum size: 320
        Compacted row maximum size: 586
        Compacted row mean size: 479

        Column Family: Payment
        SSTable count: 3
        Space used (live): 2473
        Space used (total): 2473
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 27728
        Read Latency: 0.059 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: Club
        SSTable count: 6
        Space used (live): 363609256
        Space used (total): 477968226
        Memtable Columns Count: 124134
        Memtable Data Size: 38144903
        Memtable Switch Count: 140
        Read Count: 63668
        Read Latency: 29.187 ms.
        Write Count: 1966098
        Write Latency: 0.184 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 63751
        Key cache hit rate: 0.6543073524771221
        Row cache: disabled
        Compacted row minimum size: 274
        Compacted row maximum size: 49784
        Compacted row mean size: 7711

        Column Family: RandomUsers
        SSTable count: 1
        Space used (live): 1329464
        Space used (total): 1329464
        Memtable Columns Count: 170433
        Memtable Data Size: 5525529
        Memtable Switch Count: 8
        Read Count: 58236
        Read Latency: 6.215 ms.
        Write Count: 143339
        Write Latency: 0.575 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 21
        Key cache hit rate: 1.0
        Row cache: disabled
        Compacted row minimum size: 1244196
        Compacted row maximum size: 1327409
        Compacted row mean size: 1287929

        Column Family: User
        SSTable count: 4
        Space used (live): 583812728
        Space used (total): 583812728
        Memtable Columns Count: 248833
        Memtable Data Size: 7909687
        Memtable Switch Count: 12
        Read Count: 2207667
        Read Latency: 8.413 ms.
        Write Count: 941029
        Write Latency: 0.107 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 94940
        Key cache hit rate: 0.9769078238427732
        Row cache: disabled
        Compacted row minimum size: 259
        Compacted row maximum size: 174340
        Compacted row mean size: 3916

----------------
Keyspace: Pandora
    Read Count: 1475530
    Read Latency: 5.249669284257182 ms.
    Write Count: 856550
    Write Latency: 0.16070848053236822 ms.
    Pending Tasks: 0
        Column Family: Folder
        SSTable count: 7
        Space used (live): 942926793
        Space used (total): 942926793
        Memtable Columns Count: 150119
        Memtable Data Size: 814194
        Memtable Switch Count: 21
        Read Count: 604605
        Read Latency: 7.539 ms.
        Write Count: 553015
        Write Latency: 0.173 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 111884
        Key cache hit rate: 0.9770064886992118
        Row cache: disabled
        Compacted row minimum size: 234
        Compacted row maximum size: 1644541
        Compacted row mean size: 1238

        Column Family: Attachment
        SSTable count: 5
        Space used (live): 754692822
        Space used (total): 754692822
        Memtable Columns Count: 548
        Memtable Data Size: 7110
        Memtable Switch Count: 8
        Read Count: 22708
        Read Latency: 6.950 ms.
        Write Count: 29835
        Write Latency: 0.025 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 8299
        Key cache hit rate: 0.017857142857142856
        Row cache: disabled
        Compacted row minimum size: 269
        Compacted row maximum size: 274
        Compacted row mean size: 272

        Column Family: Message
        SSTable count: 7
        Space used (live): 3689251440
        Space used (total): 3689251440
        Memtable Columns Count: 50399
        Memtable Data Size: 1361773
        Memtable Switch Count: 10
        Read Count: 163630
        Read Latency: 16.981 ms.
        Write Count: 257843
        Write Latency: 0.113 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 52427
        Key cache hit rate: 0.5041365046535677
        Row cache: disabled
        Compacted row minimum size: 272
        Compacted row maximum size: 2754
        Compacted row mean size: 752

        Column Family: FolderInfo
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 8704
        Read Latency: 0.008 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: StandardByUUID1
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 8704
        Read Latency: 0.005 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: User
        SSTable count: 4
        Space used (live): 120121309
        Space used (total): 120121309
        Memtable Columns Count: 1786
        Memtable Data Size: 50657
        Memtable Switch Count: 8
        Read Count: 641067
        Read Latency: 2.728 ms.
        Write Count: 15857
        Write Latency: 0.148 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 45965
        Key cache hit rate: 0.957460499034818
        Row cache: disabled
        Compacted row minimum size: 271
        Compacted row maximum size: 506
        Compacted row mean size: 499

        Column Family: Standard1
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 8704
        Read Latency: 0.005 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: Standard2
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 8704
        Read Latency: 0.006 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: Folder1
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 8704
        Read Latency: 0.005 ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

----------------
root@cdb-006:/glass/sfw/cassandra# bin/nodetool -h localhost tpstats
Pool Name                    Active   Pending      Completed
FILEUTILS-DELETE-POOL             0         0             70
STREAM-STAGE                      0         0              0
RESPONSE-STAGE                    0         0       18298968
ROW-READ-STAGE                    1         0       10168985
LB-OPERATIONS                     0         0              0
MESSAGE-DESERIALIZER-POOL         0         0       24960681
GMFD                              0         0          93647
LB-TARGET                         0         0              0
CONSISTENCY-MANAGER               0         0              0
ROW-MUTATION-STAGE                0         0        4280703
MESSAGE-STREAMING-POOL            0         0             12
LOAD-BALANCER-STAGE               0         0              0
FLUSH-SORTER-POOL                 0         0              0
MEMTABLE-POST-FLUSHER             0         0            224
FLUSH-WRITER-POOL                 0         0            224
AE-SERVICE-STAGE                  0         0             22
HINTED-HANDOFF-POOL               1         5             73


Mime
View raw message