cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuan Fang <y...@kryptoncloud.com>
Subject Re: Is my cluster normal?
Date Wed, 13 Jul 2016 17:32:35 GMT
$nodetool tpstats

...
Pool Name                               Active   Pending   Completed
Blocked      All time blocked
Native-Transport-Requests       128       128        1420623949         1
      142821509
...



What is this? Is it normal?

On Tue, Jul 12, 2016 at 3:03 PM, Yuan Fang <yuan@kryptoncloud.com> wrote:

> Hi Jonathan,
>
> Here is the result:
>
> ubuntu@ip-172-31-44-250:~$ iostat -dmx 2 10
> Linux 3.13.0-74-generic (ip-172-31-44-250) 07/12/2016 _x86_64_ (4 CPU)
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.01     2.13    0.74    1.55     0.01     0.02    27.77
>     0.00    0.74    0.89    0.66   0.43   0.10
> xvdf              0.01     0.58  237.41   52.50    12.90     6.21   135.02
>     2.32    8.01    3.65   27.72   0.57  16.63
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     7.50    0.00    2.50     0.00     0.04    32.00
>     0.00    1.60    0.00    1.60   1.60   0.40
> xvdf              0.00     0.00  353.50    0.00    24.12     0.00   139.75
>     0.49    1.37    1.37    0.00   0.58  20.60
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    1.00     0.00     0.00     8.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     2.00  463.50   35.00    30.69     2.86   137.84
>     0.88    1.77    1.29    8.17   0.60  30.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    1.00     0.00     0.00     8.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     0.00   99.50   36.00     8.54     4.40   195.62
>     1.55    3.88    1.45   10.61   1.06  14.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     5.00    0.00    1.50     0.00     0.03    34.67
>     0.00    1.33    0.00    1.33   1.33   0.20
> xvdf              0.00     1.50  703.00  195.00    48.83    23.76   165.57
>     6.49    8.36    1.66   32.51   0.55  49.80
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    1.00     0.00     0.04    72.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     2.50  149.50   69.50    10.12     6.68   157.14
>     0.74    3.42    1.18    8.23   0.51  11.20
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     5.00    0.00    2.50     0.00     0.03    24.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     0.00   61.50   22.50     5.36     2.75   197.64
>     0.33    3.93    1.50   10.58   0.88   7.40
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     0.00    0.00    0.50     0.00     0.00     8.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     0.00  375.00    0.00    24.84     0.00   135.64
>     0.45    1.20    1.20    0.00   0.57  21.20
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     1.00    0.00    6.00     0.00     0.03     9.33
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     0.00  542.50   23.50    35.08     2.83   137.16
>     0.80    1.41    1.15    7.23   0.49  28.00
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> xvda              0.00     3.50    0.50    1.50     0.00     0.02    24.00
>     0.00    0.00    0.00    0.00   0.00   0.00
> xvdf              0.00     1.50  272.00  153.50    16.18    18.67   167.73
>    14.32   33.66    1.39   90.84   0.81  34.60
>
>
>
> On Tue, Jul 12, 2016 at 12:34 PM, Jonathan Haddad <jon@jonhaddad.com>
> wrote:
>
>> When you have high system load it means your CPU is waiting for
>> *something*, and in my experience it's usually slow disk.  A disk connected
>> over network has been a culprit for me many times.
>>
>> On Tue, Jul 12, 2016 at 12:33 PM Jonathan Haddad <jon@jonhaddad.com>
>> wrote:
>>
>>> Can do you do:
>>>
>>> iostat -dmx 2 10
>>>
>>>
>>>
>>> On Tue, Jul 12, 2016 at 11:20 AM Yuan Fang <yuan@kryptoncloud.com>
>>> wrote:
>>>
>>>> Hi Jeff,
>>>>
>>>> The read being low is because we do not have much read operations right
>>>> now.
>>>>
>>>> The heap is only 4GB.
>>>>
>>>> MAX_HEAP_SIZE=4GB
>>>>
>>>> On Thu, Jul 7, 2016 at 7:17 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com>
>>>> wrote:
>>>>
>>>>> EBS iops scale with volume size.
>>>>>
>>>>>
>>>>>
>>>>> A 600G EBS volume only guarantees 1800 iops – if you’re exhausting
>>>>> those on writes, you’re going to suffer on reads.
>>>>>
>>>>>
>>>>>
>>>>> You have a 16G server, and probably a good chunk of that allocated to
>>>>> heap. Consequently, you have almost no page cache, so your reads are
going
>>>>> to hit the disk. Your reads being very low is not uncommon if you have
no
>>>>> page cache – the default settings for Cassandra (64k compression chunks)
>>>>> are really inefficient for small reads served off of disk. If you drop
the
>>>>> compression chunk size (4k, for example), you’ll probably see your
read
>>>>> throughput increase significantly, which will give you more iops for
>>>>> commitlog, so write throughput likely goes up, too.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From: *Jonathan Haddad <jon@jonhaddad.com>
>>>>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>>> *Date: *Thursday, July 7, 2016 at 6:54 PM
>>>>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>>>>> *Subject: *Re: Is my cluster normal?
>>>>>
>>>>>
>>>>>
>>>>> What's your CPU looking like? If it's low, check your IO with iostat
>>>>> or dstat. I know some people have used Ebs and say it's fine but ive
been
>>>>> burned too many times.
>>>>>
>>>>> On Thu, Jul 7, 2016 at 6:12 PM Yuan Fang <yuan@kryptoncloud.com>
>>>>> wrote:
>>>>>
>>>>> Hi Riccardo,
>>>>>
>>>>>
>>>>>
>>>>> Very low IO-wait. About 0.3%.
>>>>>
>>>>> No stolen CPU. It is a casssandra only instance. I did not see any
>>>>> dropped messages.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ubuntu@cassandra1:/mnt/data$ nodetool tpstats
>>>>>
>>>>> Pool Name                    Active   Pending      Completed   Blocked
>>>>>  All time blocked
>>>>>
>>>>> MutationStage                     1         1      929509244        
0
>>>>>                 0
>>>>>
>>>>> ViewMutationStage                 0         0              0        
0
>>>>>                 0
>>>>>
>>>>> ReadStage                         4         0        4021570        
0
>>>>>                 0
>>>>>
>>>>> RequestResponseStage              0         0      731477999        
0
>>>>>                 0
>>>>>
>>>>> ReadRepairStage                   0         0         165603        
0
>>>>>                 0
>>>>>
>>>>> CounterMutationStage              0         0              0        
0
>>>>>                 0
>>>>>
>>>>> MiscStage                         0         0              0        
0
>>>>>                 0
>>>>>
>>>>> CompactionExecutor                2        55          92022        
0
>>>>>                 0
>>>>>
>>>>> MemtableReclaimMemory             0         0           1736        
0
>>>>>                 0
>>>>>
>>>>> PendingRangeCalculator            0         0              6        
0
>>>>>                 0
>>>>>
>>>>> GossipStage                       0         0         345474        
0
>>>>>                 0
>>>>>
>>>>> SecondaryIndexManagement          0         0              0        
0
>>>>>                 0
>>>>>
>>>>> HintsDispatcher                   0         0              4        
0
>>>>>                 0
>>>>>
>>>>> MigrationStage                    0         0             35        
0
>>>>>                 0
>>>>>
>>>>> MemtablePostFlush                 0         0           1973        
0
>>>>>                 0
>>>>>
>>>>> ValidationExecutor                0         0              0        
0
>>>>>                 0
>>>>>
>>>>> Sampler                           0         0              0        
0
>>>>>                 0
>>>>>
>>>>> MemtableFlushWriter               0         0           1736        
0
>>>>>                 0
>>>>>
>>>>> InternalResponseStage             0         0           5311        
0
>>>>>                 0
>>>>>
>>>>> AntiEntropyStage                  0         0              0        
0
>>>>>                 0
>>>>>
>>>>> CacheCleanupExecutor              0         0              0        
0
>>>>>                 0
>>>>>
>>>>> Native-Transport-Requests       128       128      347508531        
2
>>>>>          15891862
>>>>>
>>>>>
>>>>>
>>>>> Message type           Dropped
>>>>>
>>>>> READ                         0
>>>>>
>>>>> RANGE_SLICE                  0
>>>>>
>>>>> _TRACE                       0
>>>>>
>>>>> HINT                         0
>>>>>
>>>>> MUTATION                     0
>>>>>
>>>>> COUNTER_MUTATION             0
>>>>>
>>>>> BATCH_STORE                  0
>>>>>
>>>>> BATCH_REMOVE                 0
>>>>>
>>>>> REQUEST_RESPONSE             0
>>>>>
>>>>> PAGED_RANGE                  0
>>>>>
>>>>> READ_REPAIR                  0
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 5:24 PM, Riccardo Ferrari <ferrarir@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Yuan,
>>>>>
>>>>>
>>>>>
>>>>> You machine instance is 4 vcpus that is 4 threads (not cores!!!),
>>>>> aside from any Cassandra specific discussion a system load of 10 on a
4
>>>>> threads machine is way too much in my opinion. If that is the running
>>>>> average system load I would look deeper into system details. Is that
IO
>>>>> wait? Is that CPU Stolen? Is that a Cassandra only instance or are there
>>>>> other processes pushing the load?
>>>>>
>>>>> What does your "nodetool tpstats" say? Hoe many dropped messages do
>>>>> you have?
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 8, 2016 at 12:34 AM, Yuan Fang <yuan@kryptoncloud.com>
>>>>> wrote:
>>>>>
>>>>> Thanks Ben! For the post, it seems they got a little better but
>>>>> similar result than i did. Good to know it.
>>>>>
>>>>> I am not sure if a little fine tuning of heap memory will help or
>>>>> not.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater <ben.slater@instaclustr.com>
>>>>> wrote:
>>>>>
>>>>> Hi Yuan,
>>>>>
>>>>>
>>>>>
>>>>> You might find this blog post a useful comparison:
>>>>>
>>>>>
>>>>> https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spark-and-apache-cassandra-benchmark/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_blog_2016_01_07_multi-2Ddata-2Dcenter-2Dapache-2Dspark-2Dand-2Dapache-2Dcassandra-2Dbenchmark_&d=CwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=Ltg5YUTZbI4Ixf7UjzKW636Llz6zXXurTveCLptZwio&s=MU4-NWBjvVO95HnxQtkYk4xkApq4X4IiVy8tPCgj4KU&e=>
>>>>>
>>>>>
>>>>>
>>>>> Although the focus is on Spark and Cassandra and multi-DC there are
>>>>> also some single DC benchmarks of m4.xl
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__m4.xl&d=CwQFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=Ltg5YUTZbI4Ixf7UjzKW636Llz6zXXurTveCLptZwio&s=m3DfZk3YOaf0W2OvACsqDWXp-vdlkP-cC0WnEouZwkk&e=>
>>>>> clusters plus some discussion of how we went about benchmarking.
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Ben
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 8 Jul 2016 at 07:52 Yuan Fang <yuan@kryptoncloud.com> wrote:
>>>>>
>>>>> Yes, here is my stress test result:
>>>>>
>>>>> Results:
>>>>>
>>>>> op rate                   : 12200 [WRITE:12200]
>>>>>
>>>>> partition rate            : 12200 [WRITE:12200]
>>>>>
>>>>> row rate                  : 12200 [WRITE:12200]
>>>>>
>>>>> latency mean              : 16.4 [WRITE:16.4]
>>>>>
>>>>> latency median            : 7.1 [WRITE:7.1]
>>>>>
>>>>> latency 95th percentile   : 38.1 [WRITE:38.1]
>>>>>
>>>>> latency 99th percentile   : 204.3 [WRITE:204.3]
>>>>>
>>>>> latency 99.9th percentile : 465.9 [WRITE:465.9]
>>>>>
>>>>> latency max               : 1408.4 [WRITE:1408.4]
>>>>>
>>>>> Total partitions          : 1000000 [WRITE:1000000]
>>>>>
>>>>> Total errors              : 0 [WRITE:0]
>>>>>
>>>>> total gc count            : 0
>>>>>
>>>>> total gc mb               : 0
>>>>>
>>>>> total gc time (s)         : 0
>>>>>
>>>>> avg gc time(ms)           : NaN
>>>>>
>>>>> stdev gc time(ms)         : 0
>>>>>
>>>>> Total operation time      : 00:01:21
>>>>>
>>>>> END
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla <rs@foundev.pro> wrote:
>>>>>
>>>>> Lots of variables you're leaving out.
>>>>>
>>>>>
>>>>>
>>>>> Depends on write size, if you're using logged batch or not, what
>>>>> consistency level, what RF, if the writes come in bursts, etc, etc.
>>>>> However, that's all sort of moot for determining "normal" really you
need a
>>>>> baseline as all those variables end up mattering a huge amount.
>>>>>
>>>>>
>>>>>
>>>>> I would suggest using Cassandra stress as a baseline and go from there
>>>>> depending on what those numbers say (just pick the defaults).
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>> On Jul 7, 2016, at 4:39 PM, Yuan Fang <yuan@kryptoncloud.com> wrote:
>>>>>
>>>>> yes, it is about 8k writes per node.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle <daemeonr@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Are you saying 7k writes per node? or 30k writes per node?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *.......Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang <yuan@kryptoncloud.com>
>>>>> wrote:
>>>>>
>>>>> writes 30k/second is the main thing.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle <daemeonr@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Assuming you meant 100k, that likely for something with 16mb of
>>>>> storage (probably way small) where the data is more that 64k hence will
not
>>>>> fit into the row cache.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *.......Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872
>>>>> <%28%2B44%29%20%280%29%2020%208144%209872>*
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang <yuan@kryptoncloud.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and
>>>>> 600GB ssd EBS).
>>>>>
>>>>> I can reach a cluster wide write requests of 30k/second and read
>>>>> request about 100/second. The cluster OS load constantly above 10. Are
>>>>> those normal?
>>>>>
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>>
>>>>> Yuan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> ————————
>>>>>
>>>>> Ben Slater
>>>>>
>>>>> Chief Product Officer
>>>>>
>>>>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>>>>>
>>>>> +61 437 929 798
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>

Mime
View raw message