incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From graham sanderson <gra...@vast.com>
Subject Re: Question about READS in a multi DC environment.
Date Tue, 13 May 2014 23:36:39 GMT
Yeah, but all the requests for data/digest are sent at the same time… responses that aren’t
“needed” to complete the request are dealt with asynchronously (possibly causing repair).


In the original trace (which is confusing because I don’t think the clocks are in sync)…
I don’t see anything that makes me believe it is blocking for all 3 responses - It actually
does reads on all 3 nodes even if only digests are required

On May 12, 2014, at 12:37 AM, DuyHai Doan <doanduyhai@gmail.com> wrote:

> Ins't read repair supposed to be done asynchronously in background ?
> 
> 
> On Mon, May 12, 2014 at 2:07 AM, graham sanderson <graham@vast.com> wrote:
> You have a read_repair_chance of 1.0 which is probably why your query is hitting all
data centers.
> 
> On May 11, 2014, at 3:44 PM, Mark Farnan <devmail@petrolink.com> wrote:
> 
> > Im trying to understand READ load in Cassandra across a multi-datacenter cluster.
  (Specifically why it seems to be hitting more than one DC) and hope someone can help.
> >
> > From what Iím seeing here, a READ, with Consistency LOCAL_ONE,   seems to be hitting
All 3 datacenters, rather than just the one Iím connected to.   I see  'Read 101 live and
0 tombstoned cells'  from EACH of the 3 DC"s in the trace, which seems, wrong.
> > I have tried every  Consistency level, same result.   This also is same from my
C# code via the DataStax driver, (where I first noticed the issue).
> >
> > Can someone please shed some light on what is occurring ?  Specifically I dont'
want a query on one DC, going anywhere near the other 2 as a rule, as in production,  these
DC's will be accross slower links.
> >
> >
> > Query:  (NOTE:  Whilst this uses a kairosdb table,  i'm just playing with queries
against it as it has 100k columns in this key for testing).
> >
> > cqlsh:kairosdb> consistency local_one
> > Consistency level set to LOCAL_ONE.
> >
> > cqlsh:kairosdb> select * from data_points where key = 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
limit 1000;
> >
> > ... Some return data  rows listed here which I've removed ....
> >
> > <CassandraQuery.txt>
> > Query Respose Trace:
> >
> > activity                                                                       
                                                         | timestamp    | source         |
source_elapsed
> > ------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
> >                                                                                
                                      execute_cql3_query | 07:18:12,692 | 192.168.25.111 |
             0
> >                                                                                
                   Message received from /192.168.25.111 | 07:18:00,706 | 192.168.25.131 |
            50
> >                                                                                
         Executing single-partition query on data_points | 07:18:00,707 | 192.168.25.131 |
           760
> >                                                                                
                            Acquiring sstable references | 07:18:00,707 | 192.168.25.131 |
           814
> >                                                                                
                             Merging memtable tombstones | 07:18:00,707 | 192.168.25.131 |
           924
> >                                                                                
                Bloom filter allows skipping sstable 191 | 07:18:00,707 | 192.168.25.131 |
          1050
> >                                                                                
                Bloom filter allows skipping sstable 190 | 07:18:00,707 | 192.168.25.131 |
          1166
> >                                                                                
                           Key cache hit for sstable 189 | 07:18:00,707 | 192.168.25.131 |
          1275
> >                                                                                
             Seeking to partition beginning in data file | 07:18:00,707 | 192.168.25.131 |
          1293
> >                                                                Skipped 0/3 non-slice-intersecting
sstables, included 0 due to tombstones | 07:18:00,708 | 192.168.25.131 |           2173
> >                                                                                
              Merging data from memtables and 1 sstables | 07:18:00,708 | 192.168.25.131 |
          2195
> >                                                                                
                   Read 1001 live and 0 tombstoned cells | 07:18:00,709 | 192.168.25.131 |
          3259
> >                                                                                
                   Enqueuing response to /192.168.25.111 | 07:18:00,710 | 192.168.25.131 |
          4006
> >                                                                                
                      Sending message to /192.168.25.111 | 07:18:00,710 | 192.168.25.131 |
          4210
> > Parsing select * from data_points where key = 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
limit 1000; | 07:18:12,692 | 192.168.25.111 |             52
> >                                                                                
                                     Preparing statement | 07:18:12,692 | 192.168.25.111 |
           257
> >                                                                                
                      Sending message to /192.168.25.121 | 07:18:12,693 | 192.168.25.111 |
          1099
> >                                                                                
                      Sending message to /192.168.25.131 | 07:18:12,693 | 192.168.25.111 |
          1254
> >                                                                                
         Executing single-partition query on data_points | 07:18:12,693 | 192.168.25.111 |
          1269
> >                                                                                
                            Acquiring sstable references | 07:18:12,693 | 192.168.25.111 |
          1284
> >                                                                                
                             Merging memtable tombstones | 07:18:12,694 | 192.168.25.111 |
          1315
> >                                                                                
                           Key cache hit for sstable 205 | 07:18:12,694 | 192.168.25.111 |
          1592
> >                                                                                
             Seeking to partition beginning in data file | 07:18:12,694 | 192.168.25.111 |
          1606
> >                                                                Skipped 0/1 non-slice-intersecting
sstables, included 0 due to tombstones | 07:18:12,695 | 192.168.25.111 |           2423
> >                                                                                
              Merging data from memtables and 1 sstables | 07:18:12,695 | 192.168.25.111 |
          2498
> >                                                                                
                   Read 1001 live and 0 tombstoned cells | 07:18:12,695 | 192.168.25.111 |
          3167
> >                                                                                
                   Message received from /192.168.25.121 | 07:18:12,697 | 192.168.25.111 |
          null
> >                                                                                
                Processing response from /192.168.25.121 | 07:18:12,697 | 192.168.25.111 |
          null
> >                                                                                
                   Message received from /192.168.25.131 | 07:18:12,699 | 192.168.25.111 |
          null
> >                                                                                
                Processing response from /192.168.25.131 | 07:18:12,699 | 192.168.25.111 |
          null
> >                                                                                
                   Message received from /192.168.25.111 | 07:19:49,432 | 192.168.25.121 |
            68
> >                                                                                
         Executing single-partition query on data_points | 07:19:49,433 | 192.168.25.121 |
           824
> >                                                                                
                            Acquiring sstable references | 07:19:49,433 | 192.168.25.121 |
           840
> >                                                                                
                             Merging memtable tombstones | 07:19:49,433 | 192.168.25.121 |
           898
> >                                                                                
                Bloom filter allows skipping sstable 193 | 07:19:49,433 | 192.168.25.121 |
           983
> >                                                                                
                           Key cache hit for sstable 192 | 07:19:49,433 | 192.168.25.121 |
          1055
> >                                                                                
             Seeking to partition beginning in data file | 07:19:49,433 | 192.168.25.121 |
          1073
> >                                                                Skipped 0/2 non-slice-intersecting
sstables, included 0 due to tombstones | 07:19:49,434 | 192.168.25.121 |           1803
> >                                                                                
              Merging data from memtables and 1 sstables | 07:19:49,434 | 192.168.25.121 |
          1839
> >                                                                                
                   Read 1001 live and 0 tombstoned cells | 07:19:49,434 | 192.168.25.121 |
          2518
> >                                                                                
                   Enqueuing response to /192.168.25.111 | 07:19:49,435 | 192.168.25.121 |
          3026
> >                                                                                
                      Sending message to /192.168.25.111 | 07:19:49,435 | 192.168.25.121 |
          3128
> >                                                                                
                                        Request complete | 07:18:12,696 | 192.168.25.111 |
          4387
> >
> >
> > Other Stats about the cluster:
> >
> > [root@cdev101 conf]# nodetool status
> > Datacenter: DC3
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > --  Address         Load       Tokens  Owns   Host ID                          
    Rack
> > UN  192.168.25.131  80.67 MB   256     34.2%  6ec61643-17d4-4a2e-8c44-57e08687a957
 RAC1
> > Datacenter: DC2
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > --  Address         Load       Tokens  Owns   Host ID                          
    Rack
> > UN  192.168.25.121  79.46 MB   256     30.6%  976626fb-ea80-405b-abb0-eae703b0074d
 RAC1
> > Datacenter: DC1
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > --  Address         Load       Tokens  Owns   Host ID                          
    Rack
> > UN  192.168.25.111  61.82 MB   256     35.2%  9475e2da-d926-42d0-83fb-0188d0f8f438
 RAC1
> >
> >
> > cqlsh> describe keyspace kairosdb
> >
> > CREATE KEYSPACE kairosdb WITH replication = {
> >  'class': 'NetworkTopologyStrategy',
> >  'DC2': '1',
> >  'DC3': '1',
> >  'DC1': '1'
> > };
> >
> > USE kairosdb;
> >
> > CREATE TABLE data_points (
> >  key blob,
> >  column1 blob,
> >  value blob,
> >  PRIMARY KEY (key, column1)
> > ) WITH COMPACT STORAGE AND
> >  bloom_filter_fp_chance=0.010000 AND
> >  caching='KEYS_ONLY' AND
> >  comment='' AND
> >  dclocal_read_repair_chance=0.000000 AND
> >  gc_grace_seconds=864000 AND
> >  index_interval=128 AND
> >  read_repair_chance=1.000000 AND
> >  replicate_on_write='true' AND
> >  populate_io_cache_on_flush='false' AND
> >  default_time_to_live=0 AND
> >  speculative_retry='NONE' AND
> >  memtable_flush_period_in_ms=0 AND
> >  compaction={'class': 'SizeTieredCompactionStrategy'} AND
> >  compression={'sstable_compression': 'LZ4Compressor'};
> >
> >
> >
> 
> 


Mime
View raw message