cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mohsin k <moshinkarova...@gmail.com>
Subject Re: Increased latency after setting row_cache_size_in_mb
Date Tue, 06 Feb 2018 13:50:32 GMT
Thanks, Jeff, will definitely check the trace. Also, one strange thing I
noticed, with number of threads till '64', the latency is around 3ms but as
the number of threads increases latency also increases. Eventually, at
thread count, 609 latency is around 30ms. I am using a single client to
loadtest 4 node cluster. Is this the issue because of client being
bottleneck?

On Mon, Feb 5, 2018 at 8:05 PM, Jeff Jirsa <jjirsa@gmail.com> wrote:

> Also: coordinator handles tracing and read repair. Make sure tracing is
> off for production. Have your data repaired if possible to eliminate that.
>
> Use tracing to see what’s taking the time.
>
> --
> Jeff Jirsa
>
>
> On Feb 5, 2018, at 6:32 AM, Jeff Jirsa <jjirsa@gmail.com> wrote:
>
> There’s two parts to latency on the Cassandra side:
>
> Local and coordinator
>
> When you read, the node to which you connect coordinates the request to
> the node which has the data (potentially itself). Long tail in coordinator
> latencies tend to be the coordinator itself gc’ing, which will happen from
> time to time. If it’s more consistently high, it may be natural latencies
> in your cluster (ie: your requests are going cross wan and the other dc is
> 10-20ms away).
>
> If the latency is seen in p99 but not p50, you can almost always
> speculatively read from another coordinator (driver level speculation)
> after a millisecond or so.
>
> --
> Jeff Jirsa
>
>
> On Feb 5, 2018, at 5:41 AM, mohsin k <moshinkarovalia@gmail.com> wrote:
>
> Thanks for response @Nicolas. I was considering the total read latency
> from the client to server (as shown in the image above) which is around
> 30ms. Which I want to get around 3ms (client and server are both on same
> network). I did not consider read latency provided by the server (which I
> should have). I monitored CPU , memory and JVM lifecycle, which is at a
> safe level. *I think the difference(0.03 to 30) might be because of low
> network bandwidth, correct me if I am wrong.*
>
> I did reduce chunk_length_in_kb to 4kb, but I couldn't get a considerable
> amount of difference, might be because there is less room for improvement
> on the server side.
>
> Thanks again.
>
> On Mon, Feb 5, 2018 at 6:52 PM, Nicolas Guyomar <nicolas.guyomar@gmail.com
> > wrote:
>
>> Your row hit rate is 0.971 which is already very high, IMHO there is
>> "nothing" left to do here if you can afford to store your entire dataset in
>> memory
>>
>> Local read latency: 0.030 ms already seems good to me, what makes you
>> think that you can achieve more with the relative "small" box you are using
>> ?
>>
>> You have to keep an eye on other metrics which might be a limiting
>> factor, like cpu usage, JVM heap lifecycle and so on
>>
>> For read heavy workflow it is sometimes advised to reduce chunk_length_in_kb
>> from the default 64kb to 4kb, see if it helps !
>>
>> On 5 February 2018 at 13:09, mohsin k <moshinkarovalia@gmail.com> wrote:
>>
>>> Hey Rahul,
>>>
>>> Each partition has around 10 cluster keys. Based on nodetool, I can
>>> roughly estimate partition size to be less than 1KB.
>>>
>>> On Mon, Feb 5, 2018 at 5:37 PM, mohsin k <moshinkarovalia@gmail.com>
>>> wrote:
>>>
>>>> Hey Nicolas,
>>>>
>>>> My goal is to reduce latency as much as possible. I did wait for
>>>> warmup. The test ran for more than 15mins, I am not sure why it shows 2mins
>>>> though.
>>>>
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 5:25 PM, Rahul Singh <
>>>> rahul.xavier.singh@gmail.com> wrote:
>>>>
>>>>> What is the average size of your partitions / rows. 1GB may not be
>>>>> enough.
>>>>>
>>>>> Rahul
>>>>>
>>>>> On Feb 5, 2018, 6:52 AM -0500, mohsin k <moshinkarovalia@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have been looking into different configurations for tuning my
>>>>> cassandra servers. So, initially I loadtested server using cassandra-stress
>>>>> tool, with default configs and then tuning one by one config to measure
>>>>> impact of change. First config, I tried was setting "
>>>>> *row_cache_size_in_mb*" to 1000 (MB) in yaml, adding caching {'keys':
>>>>> 'ALL', *'rows_per_partition': 'ALL'*}. After changing these configs,
>>>>> I observed that latency has increased rather than decreasing. It would
be
>>>>> really helpful if I get to understand why is this the case and what steps
>>>>> must be taken to decrease the latency.
>>>>>
>>>>> I am running a cluster with 4 nodes.
>>>>>
>>>>> Following is my schema:
>>>>>
>>>>> CREATE TABLE stresstest.user_to_segment (
>>>>>     userid text,
>>>>>     segmentid text,
>>>>>     PRIMARY KEY (userid, segmentid)
>>>>> ) WITH CLUSTERING ORDER BY (segmentid DESC)
>>>>>     AND bloom_filter_fp_chance = 0.1
>>>>>     AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
>>>>>     AND comment = 'A table to hold blog segment user relation'
>>>>>     AND compaction = {'class': 'org.apache.cassandra.db.compa
>>>>> ction.LeveledCompactionStrategy'}
>>>>>     AND compression = {'chunk_length_in_kb': '64', 'class': '
>>>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>>>     AND crc_check_chance = 1.0
>>>>>     AND dclocal_read_repair_chance = 0.1
>>>>>     AND default_time_to_live = 0
>>>>>     AND gc_grace_seconds = 864000
>>>>>     AND max_index_interval = 2048
>>>>>     AND memtable_flush_period_in_ms = 0
>>>>>     AND min_index_interval = 128
>>>>>     AND read_repair_chance = 0.0
>>>>>     AND speculative_retry = '99PERCENTILE';
>>>>>
>>>>> Following are node specs:
>>>>> RAM: 4GB
>>>>> CPU: 4 Core
>>>>> HDD: 250BG
>>>>>
>>>>>
>>>>> Following is the output of 'nodetool info' after setting
>>>>> row_cache_size_in_mb:
>>>>>
>>>>> ID                     : d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32
>>>>> Gossip active          : true
>>>>> Thrift active          : false
>>>>> Native Transport active: true
>>>>> Load                   : 10.94 MiB
>>>>> Generation No          : 1517571163
>>>>> Uptime (seconds)       : 9169
>>>>> Heap Memory (MB)       : 136.01 / 3932.00
>>>>> Off Heap Memory (MB)   : 0.10
>>>>> Data Center            : dc1
>>>>> Rack                   : rack1
>>>>> Exceptions             : 0
>>>>> Key Cache              : entries 125881, size 9.6 MiB, capacity 100
>>>>> MiB, 107 hits, 126004 requests, 0.001 recent hit rate, 14400 save period
in
>>>>> seconds
>>>>> Row Cache              : entries 125861, size 31.54 MiB, capacity 1000
>>>>> MiB, 4262684 hits, 4388545 requests, 0.971 recent hit rate, 0 save
>>>>> period in seconds
>>>>> Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0
>>>>> hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
>>>>> Chunk Cache            : entries 273, size 17.06 MiB, capacity 480
>>>>> MiB, 325 misses, 126623 requests, 0.997 recent hit rate, NaN microseconds
>>>>> miss latency
>>>>> Percent Repaired       : 100.0%
>>>>> Token                  : (invoke with -T/--tokens to see all 256
>>>>> tokens)
>>>>>
>>>>>
>>>>> Following is output of nodetool cfstats:
>>>>>
>>>>> Total number of tables: 37
>>>>> ----------------
>>>>> Keyspace : stresstest
>>>>> Read Count: 4398162
>>>>> Read Latency: 0.02184742626579012 ms.
>>>>> Write Count: 0
>>>>> Write Latency: NaN ms.
>>>>> Pending Flushes: 0
>>>>> Table: user_to_segment
>>>>> SSTable count: 1
>>>>> SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
>>>>> Space used (live): 11076103
>>>>> Space used (total): 11076103
>>>>> Space used by snapshots (total): 0
>>>>> Off heap memory used (total): 107981
>>>>> SSTable Compression Ratio: 0.5123353861375962
>>>>> Number of partitions (estimate): 125782
>>>>> Memtable cell count: 0
>>>>> Memtable data size: 0
>>>>> Memtable off heap memory used: 0
>>>>> Memtable switch count: 2
>>>>> Local read count: 4398162
>>>>> Local read latency: 0.030 ms
>>>>> Local write count: 0
>>>>> Local write latency: NaN ms
>>>>> Pending flushes: 0
>>>>> Percent repaired: 0.0
>>>>> Bloom filter false positives: 0
>>>>> Bloom filter false ratio: 0.00000
>>>>> Bloom filter space used: 79280
>>>>> Bloom filter off heap memory used: 79272
>>>>> Index summary off heap memory used: 26757
>>>>> Compression metadata off heap memory used: 1952
>>>>> Compacted partition minimum bytes: 43
>>>>> Compacted partition maximum bytes: 215
>>>>> Compacted partition mean bytes: 136
>>>>> Average live cells per slice (last five minutes): 5.719932432432432
>>>>> Maximum live cells per slice (last five minutes): 10
>>>>> Average tombstones per slice (last five minutes): 1.0
>>>>> Maximum tombstones per slice (last five minutes): 1
>>>>> Dropped Mutations: 0
>>>>>
>>>>>                  Following are my results:
>>>>>                  The blue graph is before setting
>>>>> row_cache_size_in_mb, orange is after
>>>>>
>>>>> Thanks,
>>>>> Mohsin
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message