cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3696) Adding another datacenter's node results in 0 rows returned on first datacenter
Date Thu, 05 Jan 2012 01:14:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180069#comment-13180069
] 

Brandon Williams commented on CASSANDRA-3696:
---------------------------------------------

I can reproduce this reliably, but only if read repair is on.  With it off, everything works.
 Here is failed read:

{noformat}
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 CassandraServer.java (line 323) get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 StorageProxy.java (line 603) Command/ConsistencyLevel
is SliceFromReadCommand(table='Keyspace1', key='303637', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line 90) Could not
find end point information for cassandra-1/10.179.65.102, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line 90) Could not
find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 ReadCallback.java (line 77) Blockfor/repair
is 1/true; setting up requests to /10.179.64.227,/10.179.111.137
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line 90) Could not
find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 StorageProxy.java (line 624) reading data
from /10.179.64.227
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,524 StorageProxy.java (line 644) reading digest
from /10.179.111.137
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,526 ResponseVerbHandler.java (line 44)
Processing response on a callback from 4145@/10.179.111.137
DEBUG [RequestResponseStage:10] 2012-01-05 01:00:51,526 ResponseVerbHandler.java (line 44)
Processing response on a callback from 4144@/10.179.64.227
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,526 AbstractRowResolver.java (line 66)
Preprocessed digest response
DEBUG [RequestResponseStage:10] 2012-01-05 01:00:51,526 AbstractRowResolver.java (line 66)
Preprocessed data response
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,527 PropertyFileSnitch.java (line 90) Could
not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,527 StorageProxy.java (line 672) Read: 3 ms.
{noformat}

In this case, 10.179.65.102 and 10.179.111.137 are in the first datacenter, and 10.179.64.227
is in the second.  Both DCs have an RF of one, .102 is always the coordinator and the client
is using a single connection.

For comparison, here is a successful local read where read repair did not fire (set to 10%):
{noformat}

DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 CassandraServer.java (line 323) get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 StorageProxy.java (line 603) Command/ConsistencyLevel
is SliceFromReadCommand(table='Keyspace1', key='303632', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 PropertyFileSnitch.java (line 90) Could not
find end point information for cassandra-1/10.179.65.102, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 ReadCallback.java (line 77) Blockfor/repair
is 1/false; setting up requests to cassandra-1/10.179.65.102
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 PropertyFileSnitch.java (line 90) Could not
find end point information for cassandra-1/10.179.65.102, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 StorageProxy.java (line 619) reading data
locally
DEBUG [ReadStage:93] 2012-01-05 01:00:51,520 StorageProxy.java (line 763) LocalReadRunnable
reading SliceFromReadCommand(table='Keyspace1', key='303632', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG [ReadStage:93] 2012-01-05 01:00:51,520 CollationController.java (line 192) collectAllData
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123) collecting 0
of 5: C0:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123) collecting 1
of 5: C1:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123) collecting 2
of 5: C2:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123) collecting 3
of 5: C3:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123) collecting 4
of 5: C4:false:34@1325724968371
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,522 StorageProxy.java (line 672) Read: 2 ms.
{noformat}

And one where it asked .137 for data (but did not leave the DC or repair):

{noformat}
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 CassandraServer.java (line 323) get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 StorageProxy.java (line 603) Command/ConsistencyLevel
is SliceFromReadCommand(table='Keyspace1', key='303731', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line 90) Could not
find end point information for cassandra-1/10.179.65.102, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line 90) Could not
find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line 90) Could not
find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 ReadCallback.java (line 77) Blockfor/repair
is 1/false; setting up requests to /10.179.111.137
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 PropertyFileSnitch.java (line 90) Could not
find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 StorageProxy.java (line 624) reading data
from /10.179.111.137
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517 ResponseVerbHandler.java (line 44)
Processing response on a callback from 4143@/10.179.111.137
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517 AbstractRowResolver.java (line 66)
Preprocessed data response
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517 PropertyFileSnitch.java (line 90)
Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,517 StorageProxy.java (line 672) Read: 2 ms.
{noformat}
                
> Adding another datacenter's node results in 0 rows returned on first datacenter
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3696
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3696
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Joaquin Casares
>
> On Cassandra-1.0.5:
> 1. Create a node in C* with a fresh installation and create a keyspace on that node with
one column family -
> CREATE KEYSPACE test 
> WITH placement_strategy = 'SimpleStrategy' 
> and strategy_options={replication_factor:1};
> use test; 
> create column family cf1;
> 2. Insert values into cf1 -
> set cf1[ascii('k')][ascii('c')] = ascii('v');
> get cf1[ascii('k')]; 
> => (column=63, value=v, timestamp=1325689630397000) 
> Returned 1 results.
> 3. update the strategy options from simple to networktopology with {Cassandra:1, Backup:1}

> 4. read from cf1 to make sure the options change doesn't affect anything -
> consistencylevel as LOCAL_QUORUM; 
> get cf1[ascii('k')]; 
> => (column=63, value=v, timestamp=1325689630397000) 
> Returned 1 results.
> 5. start a second node in the Backup datacenter 
> 6. read from cf1 again (on the first node) -
> consistencylevel as LOCAL_QUORUM; 
> get cf1[ascii('k')]; 
> Returned 0 results.
> After about 60 seconds, "get cf1[ascii('k')]" started to return results again. 
> Also, when running at a CL of ONE on 1.0's head, we were able to see issues as well.
> But, if more than one node was added to the second datacenter, then replication_strategy
is changed, it seems okay.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message