Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 19 Jul 2011 23:25:58 +0000 (UTC)
From: "Cathy Daw (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1863930391.5922.1311117958992.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1984424411.2382.1311036777831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Issue Comment Edited] (CASSANDRA-2918) After repair, one
 row missing from query when two rows expected
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067882#comment-13067882 ] 

Cathy Daw edited comment on CASSANDRA-2918 at 7/19/11 11:24 PM:
----------------------------------------------------------------

Interestingly, I did the same test case using the hector stress tool which writes @ CL=QUORUM.  
*Insert 1000 rows. 691 Rows Returned.*

* Bring up 3 nodes.  Create a KS with RF=3.
{code}
[default@StressKeyspace] describe keyspace StressKeyspace;  
Keyspace: StressKeyspace:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Durable Writes: true
    Options: [replication_factor:3]
  Column Families:
    ColumnFamily: StressStandard
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds: 0.0/0
      Key cache size / save period in seconds: 10000.0/14400
      Memtable thresholds: 0.2859375/1440/32 (millions of ops/MB/minutes)
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 1.0
      Replicate on write: true
      Built indexes: []
{code}

* Kill one node

*TEST #1: first two nodes don't agree*
* Insert 1000 rows with 10 columns each in batches of 100 into standard table.  
{code}
Get code from: [https://github.com/zznate/cassandra-stress]

sh bin/stress -o insert -t 2 -b 100 -n 1000 -c 10 node1:9160
{code}
* Verify Data:  
** node1: 691 rows
** node2: 1000 rows
** Nate verified the default is to write at QUORUM
* Bring up dead node. Run Repair.
* Kill two other nodes.
* Run in cli: consistencylevel as ONE; 
* Run in cli: list StressStandard limit 2000;
** node3: 691 rows

*TEST #2: 1st two nodes agree*
* Insert 1000 rows with 10 columns each in batches of 100 into standard table. Explicitly specify QUORUM. 
{code}
sh bin/stress -o insert -t 2 -b 100 -n 1000 -c 10 -L QUORUM:QUORUM node1:9160
{code}
* Verify Data:  
** node1: 1000 rows
** node2: 1000 rows
* Bring up dead node. Run Repair.
* Kill two other nodes.
* Run in cli: consistencylevel as ONE.  
* Run in cli: list StressStandard limit 2000;
** node3: 332 rows

*TEST #3*
* flush node1 and node2
* Bring up dead node - node3. Run Repair.
* Kill node1 and node2
* Run in cli: consistencylevel as ONE.  
* Run in cli: list StressStandard limit 2000;
** node3: 1000 rows


      was (Author: cdaw):
    Interestingly, I did the same test case using the hector stress tool which writes @ CL=QUORUM.  
*Insert 1000 rows. 691 Rows Returned.*

* Bring up 3 nodes.  Create a KS with RF=3.
{code}
[default@StressKeyspace] describe keyspace StressKeyspace;  
Keyspace: StressKeyspace:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Durable Writes: true
    Options: [replication_factor:3]
  Column Families:
    ColumnFamily: StressStandard
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds: 0.0/0
      Key cache size / save period in seconds: 10000.0/14400
      Memtable thresholds: 0.2859375/1440/32 (millions of ops/MB/minutes)
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 1.0
      Replicate on write: true
      Built indexes: []
{code}

* Kill one node

*TEST #1: first two nodes don't agree*
* Insert 1000 rows with 10 columns each in batches of 100 into standard table.  
{code}
sh bin/stress -o insert -t 2 -b 100 -n 1000 -c 10 node1:9160
{code}
* Verify Data:  
** node1: 691 rows
** node2: 1000 rows
** Nate verified the default is to write at QUORUM
* Bring up dead node. Run Repair.
* Kill two other nodes.
* Run in cli: consistencylevel as ONE; 
* Run in cli: list StressStandard limit 2000;
** node3: 691 rows

*TEST #2: 1st two nodes agree*
* Insert 1000 rows with 10 columns each in batches of 100 into standard table. Explicitly specify QUORUM. 
{code}
sh bin/stress -o insert -t 2 -b 100 -n 1000 -c 10 -L QUORUM:QUORUM node1:9160
{code}
* Verify Data:  
** node1: 1000 rows
** node2: 1000 rows
* Bring up dead node. Run Repair.
* Kill two other nodes.
* Run in cli: consistencylevel as ONE.  
* Run in cli: list StressStandard limit 2000;
** node3: 332 rows
  
> After repair, one row missing from query when two rows expected
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-2918
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2918
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.2
>         Environment: Cassandra-0.8 branch @ 07/18 around 1pm PST.
>            Reporter: Cathy Daw
>            Assignee: Sylvain Lebresne
>
> *Cluster Config*
> {code}
> cathy1  -  50.57.114.45 - Token: 0
> cathy2  -  50.57.107.176 - Token: 56713727820156410577229101238628035242
> cathy3  -  50.57.114.39 - Token: 113427455640312821154458202477256070484
> {code}
> *+2) Kill cathy3:  50.57.114.39+*
> {code}
> root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                113427455640312821154458202477256070484     
> 50.57.114.45    datacenter1 rack1       Up     Normal  59.84 KB        33.33%  0                                           
> 50.57.107.176   datacenter1 rack1       Up     Normal  59.85 KB        33.33%  56713727820156410577229101238628035242      
> 50.57.114.39    datacenter1 rack1       Down   Normal  59.85 KB        33.33%  113427455640312821154458202477256070484     
> {code}
> *+3) Run java stress tool+*
> {code}
> ./bin/stress -o insert -n 1000 -c 10 -l 3 -e QUORUM -d 50.57.114.45,50.57.107.176
> {code}
> *+4) Start Cassandra on cathy3+*
> *+5) Run repair on cathy3+*
> {code}
> nodetool -h cathy3 repair Keyspace1 Standard1
> {code}
> *+6) Kill cathy1 and cathy2+*
> {code}
> root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                113427455640312821154458202477256070484     
> 50.57.114.45    datacenter1 rack1       Down   Normal  105.46 KB       33.33%  0                                           
> 50.57.107.176   datacenter1 rack1       Down   Normal  106 KB          33.33%  56713727820156410577229101238628035242      
> 50.57.114.39    datacenter1 rack1       Up     Normal  331.33 KB       33.33%  113427455640312821154458202477256070484 
> {code}
> *+7) Log into cassandra-cli on cathy3 - expect 1000 rows returned+*
> {code}
> [default@Keyspace1] consistencylevel as ONE;  
> Consistency level is set to 'ONE'.
> [default@Keyspace1] list Standard1 limit 2000;
> .....
> 323 Rows Returned.
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira