incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From graham sanderson <gra...@vast.com>
Subject Re: Question about READS in a multi DC environment.
Date Mon, 12 May 2014 00:07:45 GMT
You have a read_repair_chance of 1.0 which is probably why your query is hitting all data centers.

On May 11, 2014, at 3:44 PM, Mark Farnan <devmail@petrolink.com> wrote:

> Im trying to understand READ load in Cassandra across a multi-datacenter cluster.   (Specifically
why it seems to be hitting more than one DC) and hope someone can help. 
> 
> From what Iím seeing here, a READ, with Consistency LOCAL_ONE,   seems to be hitting
All 3 datacenters, rather than just the one Iím connected to.   I see  'Read 101 live and
0 tombstoned cells'  from EACH of the 3 DC"s in the trace, which seems, wrong.     
> I have tried every  Consistency level, same result.   This also is same from my C# code
via the DataStax driver, (where I first noticed the issue). 
> 
> Can someone please shed some light on what is occurring ?  Specifically I dont' want
a query on one DC, going anywhere near the other 2 as a rule, as in production,  these DC's
will be accross slower links. 
> 
> 
> Query:  (NOTE:  Whilst this uses a kairosdb table,  i'm just playing with queries against
it as it has 100k columns in this key for testing). 
> 
> cqlsh:kairosdb> consistency local_one
> Consistency level set to LOCAL_ONE.
> 
> cqlsh:kairosdb> select * from data_points where key = 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
limit 1000;
> 
> ... Some return data  rows listed here which I've removed ....
> 
> <CassandraQuery.txt>
> Query Respose Trace: 
> 
> activity                                                                            
                                                    | timestamp    | source         | source_elapsed
> ------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
>                                                                                     
                                 execute_cql3_query | 07:18:12,692 | 192.168.25.111 |    
         0
>                                                                                     
              Message received from /192.168.25.111 | 07:18:00,706 | 192.168.25.131 |    
        50
>                                                                                     
    Executing single-partition query on data_points | 07:18:00,707 | 192.168.25.131 |    
       760
>                                                                                     
                       Acquiring sstable references | 07:18:00,707 | 192.168.25.131 |    
       814
>                                                                                     
                        Merging memtable tombstones | 07:18:00,707 | 192.168.25.131 |    
       924
>                                                                                     
           Bloom filter allows skipping sstable 191 | 07:18:00,707 | 192.168.25.131 |    
      1050
>                                                                                     
           Bloom filter allows skipping sstable 190 | 07:18:00,707 | 192.168.25.131 |    
      1166
>                                                                                     
                      Key cache hit for sstable 189 | 07:18:00,707 | 192.168.25.131 |    
      1275
>                                                                                     
        Seeking to partition beginning in data file | 07:18:00,707 | 192.168.25.131 |    
      1293
>                                                                Skipped 0/3 non-slice-intersecting
sstables, included 0 due to tombstones | 07:18:00,708 | 192.168.25.131 |           2173
>                                                                                     
         Merging data from memtables and 1 sstables | 07:18:00,708 | 192.168.25.131 |    
      2195
>                                                                                     
              Read 1001 live and 0 tombstoned cells | 07:18:00,709 | 192.168.25.131 |    
      3259
>                                                                                     
              Enqueuing response to /192.168.25.111 | 07:18:00,710 | 192.168.25.131 |    
      4006
>                                                                                     
                 Sending message to /192.168.25.111 | 07:18:00,710 | 192.168.25.131 |    
      4210
> Parsing select * from data_points where key = 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
limit 1000; | 07:18:12,692 | 192.168.25.111 |             52
>                                                                                     
                                Preparing statement | 07:18:12,692 | 192.168.25.111 |    
       257
>                                                                                     
                 Sending message to /192.168.25.121 | 07:18:12,693 | 192.168.25.111 |    
      1099
>                                                                                     
                 Sending message to /192.168.25.131 | 07:18:12,693 | 192.168.25.111 |    
      1254
>                                                                                     
    Executing single-partition query on data_points | 07:18:12,693 | 192.168.25.111 |    
      1269
>                                                                                     
                       Acquiring sstable references | 07:18:12,693 | 192.168.25.111 |    
      1284
>                                                                                     
                        Merging memtable tombstones | 07:18:12,694 | 192.168.25.111 |    
      1315
>                                                                                     
                      Key cache hit for sstable 205 | 07:18:12,694 | 192.168.25.111 |    
      1592
>                                                                                     
        Seeking to partition beginning in data file | 07:18:12,694 | 192.168.25.111 |    
      1606
>                                                                Skipped 0/1 non-slice-intersecting
sstables, included 0 due to tombstones | 07:18:12,695 | 192.168.25.111 |           2423
>                                                                                     
         Merging data from memtables and 1 sstables | 07:18:12,695 | 192.168.25.111 |    
      2498
>                                                                                     
              Read 1001 live and 0 tombstoned cells | 07:18:12,695 | 192.168.25.111 |    
      3167
>                                                                                     
              Message received from /192.168.25.121 | 07:18:12,697 | 192.168.25.111 |    
      null
>                                                                                     
           Processing response from /192.168.25.121 | 07:18:12,697 | 192.168.25.111 |    
      null
>                                                                                     
              Message received from /192.168.25.131 | 07:18:12,699 | 192.168.25.111 |    
      null
>                                                                                     
           Processing response from /192.168.25.131 | 07:18:12,699 | 192.168.25.111 |    
      null
>                                                                                     
              Message received from /192.168.25.111 | 07:19:49,432 | 192.168.25.121 |    
        68
>                                                                                     
    Executing single-partition query on data_points | 07:19:49,433 | 192.168.25.121 |    
       824
>                                                                                     
                       Acquiring sstable references | 07:19:49,433 | 192.168.25.121 |    
       840
>                                                                                     
                        Merging memtable tombstones | 07:19:49,433 | 192.168.25.121 |    
       898
>                                                                                     
           Bloom filter allows skipping sstable 193 | 07:19:49,433 | 192.168.25.121 |    
       983
>                                                                                     
                      Key cache hit for sstable 192 | 07:19:49,433 | 192.168.25.121 |    
      1055
>                                                                                     
        Seeking to partition beginning in data file | 07:19:49,433 | 192.168.25.121 |    
      1073
>                                                                Skipped 0/2 non-slice-intersecting
sstables, included 0 due to tombstones | 07:19:49,434 | 192.168.25.121 |           1803
>                                                                                     
         Merging data from memtables and 1 sstables | 07:19:49,434 | 192.168.25.121 |    
      1839
>                                                                                     
              Read 1001 live and 0 tombstoned cells | 07:19:49,434 | 192.168.25.121 |    
      2518
>                                                                                     
              Enqueuing response to /192.168.25.111 | 07:19:49,435 | 192.168.25.121 |    
      3026
>                                                                                     
                 Sending message to /192.168.25.111 | 07:19:49,435 | 192.168.25.121 |    
      3128
>                                                                                     
                                   Request complete | 07:18:12,696 | 192.168.25.111 |    
      4387
> 
> 
> Other Stats about the cluster:
> 
> [root@cdev101 conf]# nodetool status
> Datacenter: DC3
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Tokens  Owns   Host ID                               Rack
> UN  192.168.25.131  80.67 MB   256     34.2%  6ec61643-17d4-4a2e-8c44-57e08687a957  RAC1
> Datacenter: DC2
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Tokens  Owns   Host ID                               Rack
> UN  192.168.25.121  79.46 MB   256     30.6%  976626fb-ea80-405b-abb0-eae703b0074d  RAC1
> Datacenter: DC1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Tokens  Owns   Host ID                               Rack
> UN  192.168.25.111  61.82 MB   256     35.2%  9475e2da-d926-42d0-83fb-0188d0f8f438  RAC1
> 
> 
> cqlsh> describe keyspace kairosdb
> 
> CREATE KEYSPACE kairosdb WITH replication = {
>  'class': 'NetworkTopologyStrategy',
>  'DC2': '1',
>  'DC3': '1',
>  'DC1': '1'
> };
> 
> USE kairosdb;
> 
> CREATE TABLE data_points (
>  key blob,
>  column1 blob,
>  value blob,
>  PRIMARY KEY (key, column1)
> ) WITH COMPACT STORAGE AND
>  bloom_filter_fp_chance=0.010000 AND
>  caching='KEYS_ONLY' AND
>  comment='' AND
>  dclocal_read_repair_chance=0.000000 AND
>  gc_grace_seconds=864000 AND
>  index_interval=128 AND
>  read_repair_chance=1.000000 AND
>  replicate_on_write='true' AND
>  populate_io_cache_on_flush='false' AND
>  default_time_to_live=0 AND
>  speculative_retry='NONE' AND
>  memtable_flush_period_in_ms=0 AND
>  compaction={'class': 'SizeTieredCompactionStrategy'} AND
>  compression={'sstable_compression': 'LZ4Compressor'};
> 
> 
> 


Mime
View raw message