cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 曹志富 <cao.zh...@gmail.com>
Subject Re: terrible read/write latency fluctuation
Date Sun, 25 Oct 2015 02:54:14 GMT
beauty trace log :
[image: 内嵌图片 1]

--------------------------------------
Ranger Tsao

2015-10-25 10:51 GMT+08:00 曹志富 <cao.zhifu@gmail.com>:

> schema is simple  :
> CREATE TABLE k1.c1 (
>     key text,
>     column1 text,
>     value text,
>     PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE
>     AND CLUSTERING ORDER BY (column1 ASC)
>     AND bloom_filter_fp_chance = 0.01
>     AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>     AND comment = ''
>     AND compaction = {'min_threshold': '4',
> 'unchecked_tombstone_compaction': 'false', 'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32'}
>     AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 0
>     AND gc_grace_seconds = 0
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99.0PERCENTILE';
>
>
> One row key 25 columns.
>
> 25 nodes ,RF 3 ,one node 1TB.
>
> without superuser login cqlsh ,can't print trace log :
>
>
> Tracing session: 20a7fe00-7ac3-11e5-ad2a-0bbf95bc6f42
>
>  activity
>                         | timestamp                  | source        |
> source_elapsed
>
> --------------------------------------------------------------------------------------------------+----------------------------+---------------+----------------
>
>      Execute CQL3 query | 2015-10-25 10:50:18.976000 |  172.19.105.9 |
>          0
>                READ message received from /172.19.105.9
> [MessagingService-Incoming-/172.19.105.9] | 2015-10-25 10:50:18.977000 |
> 172.19.105.53 |             50
>    Parsing select * from weibo.content1 where key =
> 'tw650663253366837248'; [SharedPool-Worker-5] | 2015-10-25 10:50:18.977000
> |  172.19.105.9 |             75
>                                   Executing single-partition query on
> users [SharedPool-Worker-1] | 2015-10-25 10:50:18.978000 | 172.19.105.53 |
>           1179
>                                                         Preparing
> statement [SharedPool-Worker-5] | 2015-10-25 10:50:18.978000 |
>  172.19.105.9 |            113
>                                                Acquiring sstable
> references [SharedPool-Worker-1] | 2015-10-25 10:50:18.978000 |
> 172.19.105.53 |           1218
>                                            reading data from /
> 172.19.105.53 [SharedPool-Worker-5] | 2015-10-25 10:50:18.978000 |
>  172.19.105.9 |            542
>                 Sending READ message to /172.19.105.53
> [MessagingService-Outgoing-/172.19.105.53] | 2015-10-25 10:50:18.978000 |
>  172.19.105.9 |            619
>                                                 Merging memtable
> tombstones [SharedPool-Worker-1] | 2015-10-25 10:50:18.979000 |
> 172.19.105.53 |           1249
>                READ message received from /172.19.105.9
> [MessagingService-Incoming-/172.19.105.9] | 2015-10-25 10:50:18.979000 |
> 172.19.105.58 |             37
>                                    speculating read retry on /
> 172.19.105.71 [SharedPool-Worker-5] | 2015-10-25 10:50:18.979000 |
>  172.19.105.9 |           2794
>                                                 Key cache hit for sstable
> 1 [SharedPool-Worker-1] | 2015-10-25 10:50:18.979000 | 172.19.105.53 |
>       1291
>                 Sending READ message to /172.19.105.71
> [MessagingService-Outgoing-/172.19.105.71] | 2015-10-25 10:50:18.979000 |
>  172.19.105.9 |           2862
>                                 Seeking to partition beginning in data
> file [SharedPool-Worker-1] | 2015-10-25 10:50:18.979000 | 172.19.105.53 |
>         1301
>  REQUEST_RESPONSE message received from /172.19.105.53
> [MessagingService-Incoming-/172.19.105.53] | 2015-10-25 10:50:18.979000 |
>  172.19.105.9 |           3028
>                                                 Key cache hit for sstable
> 2 [SharedPool-Worker-1] | 2015-10-25 10:50:18.980000 | 172.19.105.53 |
>       1346
>                                Executing single-partition query on
> content1 [SharedPool-Worker-2] | 2015-10-25 10:50:18.980000 | 172.19.105.58
> |            125
>                                     Processing response from /
> 172.19.105.53 [SharedPool-Worker-1] | 2015-10-25 10:50:18.980000 |
>  172.19.105.9 |           3081
>                                 Seeking to partition beginning in data
> file [SharedPool-Worker-1] | 2015-10-25 10:50:18.980000 | 172.19.105.53 |
>         1356
>                                                Acquiring sstable
> references [SharedPool-Worker-2] | 2015-10-25 10:50:18.980000 |
> 172.19.105.58 |            131
>                                            reading data from /
> 172.19.105.58 [SharedPool-Worker-5] | 2015-10-25 10:50:18.980000 |
>  172.19.105.9 |           3196
>                                                 Merging memtable
> tombstones [SharedPool-Worker-2] | 2015-10-25 10:50:18.980000 |
> 172.19.105.58 |            140
>                 Sending READ message to /172.19.105.58
> [MessagingService-Outgoing-/172.19.105.58] | 2015-10-25 10:50:18.980000 |
>  172.19.105.9 |           3252
>   Skipped 0/2 non-slice-intersecting sstables, included 0 due to
> tombstones [SharedPool-Worker-1] | 2015-10-25 10:50:18.981000 |
> 172.19.105.53 |           1385
>                                             Key cache hit for sstable
> 66818 [SharedPool-Worker-2] | 2015-10-25 10:50:18.981000 | 172.19.105.58 |
>            185
>  REQUEST_RESPONSE message received from /172.19.105.58
> [MessagingService-Incoming-/172.19.105.58] | 2015-10-25 10:50:18.981000 |
>  172.19.105.9 |           4343
>                                  Merging data from memtables and 2
> sstables [SharedPool-Worker-1] | 2015-10-25 10:50:18.981000 | 172.19.105.53
> |           1392
>                                 Seeking to partition beginning in data
> file [SharedPool-Worker-2] | 2015-10-25 10:50:18.981000 | 172.19.105.58 |
>          189
>                                     Processing response from /
> 172.19.105.58 [SharedPool-Worker-3] | 2015-10-25 10:50:18.981000 |
>  172.19.105.9 |           4444
>                                           Read 1 live and 0 tombstone
> cells [SharedPool-Worker-1] | 2015-10-25 10:50:18.981000 | 172.19.105.53 |
>           1442
>   Skipped 0/1 non-slice-intersecting sstables, included 0 due to
> tombstones [SharedPool-Worker-2] | 2015-10-25 10:50:18.981000 |
> 172.19.105.58 |            394
>                                         Enqueuing response to /
> 172.19.105.9 [SharedPool-Worker-1] | 2015-10-25 10:50:18.982000 |
> 172.19.105.53 |           1548
>                                  Merging data from memtables and 1
> sstables [SharedPool-Worker-2] | 2015-10-25 10:50:18.982000 | 172.19.105.58
> |            406
>       Sending REQUEST_RESPONSE message to /172.19.105.9
> [MessagingService-Outgoing-/172.19.105.9] | 2015-10-25 10:50:18.982000 |
> 172.19.105.53 |           1640
>                                          Read 23 live and 0 tombstone
> cells [SharedPool-Worker-2] | 2015-10-25 10:50:18.982000 | 172.19.105.58 |
>            438
>                                         Enqueuing response to /
> 172.19.105.9 [SharedPool-Worker-2] | 2015-10-25 10:50:18.982000 |
> 172.19.105.58 |            509
>       Sending REQUEST_RESPONSE message to /172.19.105.9
> [MessagingService-Outgoing-/172.19.105.9] | 2015-10-25 10:50:18.983000 |
> 172.19.105.58 |            557
>
>        Request complete | 2015-10-25 10:50:18.980520 |  172.19.105.9 |
>       4520
>
>
>
>
> --------------------------------------
> Ranger Tsao
>
> 2015-10-25 3:40 GMT+08:00 Dor Laor <dor@scylladb.com>:
>
>> Can you provide more details - the schema, the config, especially
>> compaction policy,
>> amount of data and eventually cql trace of the reads.
>>
>> Best,
>> Dor
>>
>> On Fri, Oct 23, 2015 at 2:16 PM, 曹志富 <cao.zhifu@gmail.com> wrote:
>>
>>> additional info:
>>> 25nodes,4000 write requests ,900 read requests the whole cluster.
>>>
>>> --------------------------------------
>>> Ranger Tsao
>>>
>>> 2015-10-23 19:12 GMT+08:00 曹志富 <cao.zhifu@gmail.com>:
>>>
>>>> some nodes seems normal ,but some nodes do't. here is the metrics from
>>>> opscenter:
>>>> [image: 内嵌图片 1]
>>>>
>>>> [image: 内嵌图片 2]
>>>>
>>>> [image: 内嵌图片 3]
>>>>
>>>>
>>>> --------------------------------------
>>>> Ranger Tsao
>>>>
>>>
>>>
>>
>

Mime
View raw message