cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cathy Daw (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-2918) After repair, one row missing from query when two rows expected
Date Tue, 19 Jul 2011 01:12:57 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cathy Daw updated CASSANDRA-2918:
---------------------------------

    Description: 
*Cluster Config*
{code}
cathy1  -  50.57.114.45 - Token: 0
cathy2  -  50.57.107.176 - Token: 56713727820156410577229101238628035242
cathy3  -  50.57.114.39 - Token: 113427455640312821154458202477256070484
{code}


*+1) Create Seed Data+*
{code}
create keyspace testKS with placement_strategy = 'SimpleStrategy' and strategy_options = [{replication_factor : 3}];
{code}


*+2) Kill cathy3:  50.57.114.39+*
{code}
root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring

Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               113427455640312821154458202477256070484     
50.57.114.45    datacenter1 rack1       Up     Normal  59.84 KB        33.33%  0                                           
50.57.107.176   datacenter1 rack1       Up     Normal  59.85 KB        33.33%  56713727820156410577229101238628035242      
50.57.114.39    datacenter1 rack1       Down   Normal  59.85 KB        33.33%  113427455640312821154458202477256070484     
{code}


*+3) Run cassandra-cli+*
{code}
use testKS;
create column family metadataCF 
with key_validation_class = 'AsciiType' 
and comparator = 'AsciiType' 
and column_metadata = [
{column_name: ascii_col, validation_class:AsciiType, index_type: KEYS},
{column_name: byte_col, validation_class: BytesType, index_type: KEYS},
{column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS},
{column_name: int_col, validation_class: IntegerType, index_type: KEYS},
{column_name: long_col, validation_class: LongType, index_type: KEYS},
{column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}];

set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii column');
set metadataCF['key1']['byte_col']=bytes('10101010');
set metadataCF['key1']['uuid_col']=timeuuid();
set metadataCF['key1']['int_col']=integer(1000);
set metadataCF['key1']['long_col']=long(44444444444);
set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8 column');

//Please note: forget to change CL before inserting 'key1', so that was inserted with CL=ONE by default.
consistencylevel as TWO;

set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into ascii column');
set metadataCF['key2']['byte_col']=bytes('201010102');
set metadataCF['key2']['uuid_col']=timeuuid();
set metadataCF['key2']['int_col']=integer(2000);
set metadataCF['key2']['long_col']=long(222222222);
set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8 column');

//Assumed that the following read would be done on CL=TWO and will the second replica would be guaranteed to be fix for 'key1'.
list metadataCF;
{code}

{code}
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}


*+4) Start Cassandra on cathy3+*


*+5) Run repair on cathy3+*
{code}
nodetool -h cathy3 repair testKS metadataCF
{code}

{code}
 INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>, #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>]
 INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>, #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>]
 INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>, #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>]
 INFO 00:38:33,052 Sending AEService tree for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,100 Sending AEService tree for #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
 INFO 00:38:33,203 Sending AEService tree for #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>
 INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
 INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
 INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
 INFO 00:38:33,364 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 1 outstanding.
 INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
 INFO 00:38:33,429 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 1 outstanding.
 INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
 INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
 INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
 INFO 00:38:33,519 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 1 outstanding.
 INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
 INFO 00:38:33,555 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 0 outstanding.
 INFO 00:38:33,555 Repair session manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully.
 INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
 INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s) out of sync for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
 INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,626 Stream context metadata [], 0 sstables.
 INFO 00:38:33,643 Streaming to /50.57.114.45
 INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1
 INFO 00:38:33,736 Submitting index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db')
 INFO 00:38:33,741 Enqueuing flush of Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,748 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114 bytes)
 INFO 00:38:33,750 Enqueuing flush of Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,756 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes)
 INFO 00:38:33,758 Enqueuing flush of Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,820 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes)
 INFO 00:38:33,821 Enqueuing flush of Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,828 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes)
 INFO 00:38:33,829 Enqueuing flush of Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,836 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes)
 INFO 00:38:33,837 Enqueuing flush of Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,857 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes)
 INFO 00:38:33,858 Index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete
 INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45
 INFO 00:38:33,859 Finished streaming repair for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,859 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 0 outstanding.
 INFO 00:38:33,859 Repair session manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully.
 INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
 INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
 INFO 00:38:36,058 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 0 outstanding.
 INFO 00:38:36,058 Repair session manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully.
{code}


*+6) Kill cathy1 and cathy2+*
{code}
root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               113427455640312821154458202477256070484     
50.57.114.45    datacenter1 rack1       Down   Normal  105.46 KB       33.33%  0                                           
50.57.107.176   datacenter1 rack1       Down   Normal  106 KB          33.33%  56713727820156410577229101238628035242      
50.57.114.39    datacenter1 rack1       Up     Normal  331.33 KB       33.33%  113427455640312821154458202477256070484 
{code}


*+7) Log into cassandra-cli on node3+*
{code}
 ./cassandra-cli -h cathy3
Connected to: "Test Cluster" on cathy3/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] consistencylevel as ONE;
Consistency level is set to 'ONE'.
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}


*+8) Bring up nodes 1 and 2 and query+*
*NODE1 - cassandra-cli*
{code}
./cassandra-cli -h cathy1
Connected to: "Test Cluster" on cathy1/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}

*NODE2 - cassandra-cli*
{code}
./cassandra-cli -h cathy2
Connected to: "Test Cluster" on cathy2/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}


*NODE3 - cassandra-cli*
{code}
./cassandra-cli -h cathy3
Connected to: "Test Cluster" on cathy3/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}


*+9) Force Read Repair on node1+*
{code}
consistencylevel as THREE;
Consistency level is set to 'THREE'.
[default@testKS] list metadataCF;          
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}

  was:
*Cluster Config*
{code}
cathy1  -  50.57.114.45 - Token: 0
cathy2  -  50.57.107.176 - Token: 56713727820156410577229101238628035242
cathy3  -  50.57.114.39 - Token: 113427455640312821154458202477256070484
{code}


*+1) Create Seed Data+*
{code}
create keyspace testKS with placement_strategy = 'SimpleStrategy' and strategy_options = [{replication_factor : 3}];
{code}


*+2) Kill cathy3:  50.57.114.39+*
{code}
root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring

Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               113427455640312821154458202477256070484     
50.57.114.45    datacenter1 rack1       Up     Normal  59.84 KB        33.33%  0                                           
50.57.107.176   datacenter1 rack1       Up     Normal  59.85 KB        33.33%  56713727820156410577229101238628035242      
50.57.114.39    datacenter1 rack1       Down   Normal  59.85 KB        33.33%  113427455640312821154458202477256070484     
{code}


*+3) Run cassandra-cli+*
{code}
use testKS;
create column family metadataCF 
with key_validation_class = 'AsciiType' 
and comparator = 'AsciiType' 
and column_metadata = [
{column_name: ascii_col, validation_class:AsciiType, index_type: KEYS},
{column_name: byte_col, validation_class: BytesType, index_type: KEYS},
{column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS},
{column_name: int_col, validation_class: IntegerType, index_type: KEYS},
{column_name: long_col, validation_class: LongType, index_type: KEYS},
{column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}];

set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii column');
set metadataCF['key1']['byte_col']=bytes('10101010');
set metadataCF['key1']['uuid_col']=timeuuid();
set metadataCF['key1']['int_col']=integer(1000);
set metadataCF['key1']['long_col']=long(44444444444);
set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8 column');

consistencylevel as TWO;

set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into ascii column');
set metadataCF['key2']['byte_col']=bytes('201010102');
set metadataCF['key2']['uuid_col']=timeuuid();
set metadataCF['key2']['int_col']=integer(2000);
set metadataCF['key2']['long_col']=long(222222222);
set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8 column');
{code}

{code}
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}


*+4) Start Cassandra on cathy3+*


*+5) Run repair on cathy3+*
{code}
nodetool -h cathy3 repair testKS metadataCF
{code}

{code}
 INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>, #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>]
 INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>, #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>]
 INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>, #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>]
 INFO 00:38:33,052 Sending AEService tree for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,100 Sending AEService tree for #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
 INFO 00:38:33,203 Sending AEService tree for #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>
 INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
 INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
 INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
 INFO 00:38:33,364 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 1 outstanding.
 INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
 INFO 00:38:33,429 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 1 outstanding.
 INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
 INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
 INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
 INFO 00:38:33,519 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 1 outstanding.
 INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
 INFO 00:38:33,555 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 0 outstanding.
 INFO 00:38:33,555 Repair session manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully.
 INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
 INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s) out of sync for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
 INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,626 Stream context metadata [], 0 sstables.
 INFO 00:38:33,643 Streaming to /50.57.114.45
 INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1
 INFO 00:38:33,736 Submitting index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db')
 INFO 00:38:33,741 Enqueuing flush of Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,748 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114 bytes)
 INFO 00:38:33,750 Enqueuing flush of Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,756 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes)
 INFO 00:38:33,758 Enqueuing flush of Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,820 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes)
 INFO 00:38:33,821 Enqueuing flush of Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,828 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes)
 INFO 00:38:33,829 Enqueuing flush of Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,836 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes)
 INFO 00:38:33,837 Enqueuing flush of Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
 INFO 00:38:33,857 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes)
 INFO 00:38:33,858 Index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete
 INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45
 INFO 00:38:33,859 Finished streaming repair for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
 INFO 00:38:33,859 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 0 outstanding.
 INFO 00:38:33,859 Repair session manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully.
 INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
 INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
 INFO 00:38:36,058 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 0 outstanding.
 INFO 00:38:36,058 Repair session manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully.
{code}


*+6) Kill cathy1 and cathy2+*
{code}
root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               113427455640312821154458202477256070484     
50.57.114.45    datacenter1 rack1       Down   Normal  105.46 KB       33.33%  0                                           
50.57.107.176   datacenter1 rack1       Down   Normal  106 KB          33.33%  56713727820156410577229101238628035242      
50.57.114.39    datacenter1 rack1       Up     Normal  331.33 KB       33.33%  113427455640312821154458202477256070484 
{code}


*+7) Log into cassandra-cli on node3+*
{code}
 ./cassandra-cli -h cathy3
Connected to: "Test Cluster" on cathy3/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] consistencylevel as ONE;
Consistency level is set to 'ONE'.
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}


*+8) Bring up nodes 1 and 2 and query+*
*NODE1 - cassandra-cli*
{code}
./cassandra-cli -h cathy1
Connected to: "Test Cluster" on cathy1/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}

*NODE2 - cassandra-cli*
{code}
./cassandra-cli -h cathy2
Connected to: "Test Cluster" on cathy2/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}


*NODE3 - cassandra-cli*
{code}
./cassandra-cli -h cathy3
Connected to: "Test Cluster" on cathy3/9160
Welcome to the Cassandra CLI.

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown] use testKS;
Authenticated to keyspace: testKS
[default@testKS] list metadataCF;
Using default limit of 100
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

1 Row Returned.
{code}


*+9) Force Read Repair on node1+*
{code}
consistencylevel as THREE;
Consistency level is set to 'THREE'.
[default@testKS] list metadataCF;          
Using default limit of 100
-------------------
RowKey: key1
=> (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
=> (column=byte_col, value=10101010, timestamp=1311035260662000)
=> (column=int_col, value=1000, timestamp=1311035260669000)
=> (column=long_col, value=44444444444, timestamp=1311035260674000)
=> (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
=> (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
-------------------
RowKey: key2
=> (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
=> (column=byte_col, value=0201010102, timestamp=1311035260685000)
=> (column=int_col, value=2000, timestamp=1311035260692000)
=> (column=long_col, value=222222222, timestamp=1311035260695000)
=> (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
=> (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)

2 Rows Returned.
{code}


> After repair, one row missing from query when two rows expected
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-2918
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2918
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.2
>         Environment: Cassandra-0.8 branch @ 07/18 around 1pm PST.
>            Reporter: Cathy Daw
>
> *Cluster Config*
> {code}
> cathy1  -  50.57.114.45 - Token: 0
> cathy2  -  50.57.107.176 - Token: 56713727820156410577229101238628035242
> cathy3  -  50.57.114.39 - Token: 113427455640312821154458202477256070484
> {code}
> *+1) Create Seed Data+*
> {code}
> create keyspace testKS with placement_strategy = 'SimpleStrategy' and strategy_options = [{replication_factor : 3}];
> {code}
> *+2) Kill cathy3:  50.57.114.39+*
> {code}
> root@cathy2:~/cass-0.8/bin# ./nodetool -h localhost ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                113427455640312821154458202477256070484     
> 50.57.114.45    datacenter1 rack1       Up     Normal  59.84 KB        33.33%  0                                           
> 50.57.107.176   datacenter1 rack1       Up     Normal  59.85 KB        33.33%  56713727820156410577229101238628035242      
> 50.57.114.39    datacenter1 rack1       Down   Normal  59.85 KB        33.33%  113427455640312821154458202477256070484     
> {code}
> *+3) Run cassandra-cli+*
> {code}
> use testKS;
> create column family metadataCF 
> with key_validation_class = 'AsciiType' 
> and comparator = 'AsciiType' 
> and column_metadata = [
> {column_name: ascii_col, validation_class:AsciiType, index_type: KEYS},
> {column_name: byte_col, validation_class: BytesType, index_type: KEYS},
> {column_name: uuid_col, validation_class: LexicalUUIDType, index_type: KEYS},
> {column_name: int_col, validation_class: IntegerType, index_type: KEYS},
> {column_name: long_col, validation_class: LongType, index_type: KEYS},
> {column_name: utf8_col, validation_class: UTF8Type, index_type: KEYS}];
> set metadataCF['key1']['ascii_col']=ascii('this is data inserted into ascii column');
> set metadataCF['key1']['byte_col']=bytes('10101010');
> set metadataCF['key1']['uuid_col']=timeuuid();
> set metadataCF['key1']['int_col']=integer(1000);
> set metadataCF['key1']['long_col']=long(44444444444);
> set metadataCF['key1']['utf8_col']=utf8('this is data inserted into UTF8 column');
> //Please note: forget to change CL before inserting 'key1', so that was inserted with CL=ONE by default.
> consistencylevel as TWO;
> set metadataCF['key2']['ascii_col']=ascii('key2: this is data inserted into ascii column');
> set metadataCF['key2']['byte_col']=bytes('201010102');
> set metadataCF['key2']['uuid_col']=timeuuid();
> set metadataCF['key2']['int_col']=integer(2000);
> set metadataCF['key2']['long_col']=long(222222222);
> set metadataCF['key2']['utf8_col']=utf8('key2-this is data inserted into UTF8 column');
> //Assumed that the following read would be done on CL=TWO and will the second replica would be guaranteed to be fix for 'key1'.
> list metadataCF;
> {code}
> {code}
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 2 Rows Returned.
> {code}
> *+4) Start Cassandra on cathy3+*
> *+5) Run repair on cathy3+*
> {code}
> nodetool -h cathy3 repair testKS metadataCF
> {code}
> {code}
>  INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>, #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>]
>  INFO 00:38:32,696 Waiting for repair requests: [#<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>, #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>]
>  INFO 00:38:32,697 Waiting for repair requests: [#<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>, #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>]
>  INFO 00:38:33,052 Sending AEService tree for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.39, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
>  INFO 00:38:33,100 Sending AEService tree for #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.39, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>
>  INFO 00:38:33,203 Sending AEService tree for #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.39, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>
>  INFO 00:38:33,242 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
>  INFO 00:38:33,270 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
>  INFO 00:38:33,363 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
>  INFO 00:38:33,364 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.107.176, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 1 outstanding.
>  INFO 00:38:33,428 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
>  INFO 00:38:33,429 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.114.45, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 1 outstanding.
>  INFO 00:38:33,441 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
>  INFO 00:38:33,453 Queueing comparison #<Differencer #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]>>
>  INFO 00:38:33,519 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
>  INFO 00:38:33,519 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.107.176, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 1 outstanding.
>  INFO 00:38:33,554 Endpoints /50.57.114.39 and /50.57.114.45 are consistent for (testKS,metadataCF) on (56713727820156410577229101238628035242,113427455640312821154458202477256070484]
>  INFO 00:38:33,555 #<TreeRequest manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc, /50.57.114.45, (testKS,metadataCF), (56713727820156410577229101238628035242,113427455640312821154458202477256070484]> completed successfully: 0 outstanding.
>  INFO 00:38:33,555 Repair session manual-repair-20f203a7-53b1-4eea-a9bd-4cbaa7b434dc completed successfully.
>  INFO 00:38:33,567 Queueing comparison #<Differencer #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>>
>  INFO 00:38:33,598 Endpoints /50.57.114.39 and /50.57.114.45 have 1 range(s) out of sync for (testKS,metadataCF) on (113427455640312821154458202477256070484,0]
>  INFO 00:38:33,599 Performing streaming repair of 1 ranges for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
>  INFO 00:38:33,626 Stream context metadata [], 0 sstables.
>  INFO 00:38:33,643 Streaming to /50.57.114.45
>  INFO 00:38:33,732 Opening /var/lib/cassandra/data/testKS/metadataCF-g-1
>  INFO 00:38:33,736 Submitting index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, for data in SSTableReader(path='/var/lib/cassandra/data/testKS/metadataCF-g-1-Data.db')
>  INFO 00:38:33,741 Enqueuing flush of Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,742 Writing Memtable-metadataCF.ascii_col_idx@1146840108(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,748 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.ascii_col_idx-g-1-Data.db (114 bytes)
>  INFO 00:38:33,750 Enqueuing flush of Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,750 Writing Memtable-metadataCF.byte_col_idx@1562533825(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,756 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.byte_col_idx-g-1-Data.db (74 bytes)
>  INFO 00:38:33,758 Enqueuing flush of Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,758 Writing Memtable-metadataCF.int_col_idx@1515686296(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,820 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.int_col_idx-g-1-Data.db (71 bytes)
>  INFO 00:38:33,821 Enqueuing flush of Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,821 Writing Memtable-metadataCF.long_col_idx@507274563(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,828 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.long_col_idx-g-1-Data.db (77 bytes)
>  INFO 00:38:33,829 Enqueuing flush of Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,830 Writing Memtable-metadataCF.utf8_col_idx@1201340161(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,836 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.utf8_col_idx-g-1-Data.db (112 bytes)
>  INFO 00:38:33,837 Enqueuing flush of Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,838 Writing Memtable-metadataCF.uuid_col_idx@703489457(19/23 serialized/live bytes, 1 ops)
>  INFO 00:38:33,857 Completed flushing /var/lib/cassandra/data/testKS/metadataCF.uuid_col_idx-g-1-Data.db (85 bytes)
>  INFO 00:38:33,858 Index build of ascii_col,byte_col,int_col,long_col,utf8_col,uuid_col, complete
>  INFO 00:38:33,858 Finished streaming session 4937677751163 from /50.57.114.45
>  INFO 00:38:33,859 Finished streaming repair for #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]>
>  INFO 00:38:33,859 #<TreeRequest manual-repair-9a957823-9969-4b91-900a-96a00659a85e, /50.57.114.45, (testKS,metadataCF), (113427455640312821154458202477256070484,0]> completed successfully: 0 outstanding.
>  INFO 00:38:33,859 Repair session manual-repair-9a957823-9969-4b91-900a-96a00659a85e completed successfully.
>  INFO 00:38:36,026 Queueing comparison #<Differencer #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]>>
>  INFO 00:38:36,057 Endpoints /50.57.114.39 and /50.57.107.176 are consistent for (testKS,metadataCF) on (0,56713727820156410577229101238628035242]
>  INFO 00:38:36,058 #<TreeRequest manual-repair-9b989c5e-66a1-459d-93ed-793acc401794, /50.57.107.176, (testKS,metadataCF), (0,56713727820156410577229101238628035242]> completed successfully: 0 outstanding.
>  INFO 00:38:36,058 Repair session manual-repair-9b989c5e-66a1-459d-93ed-793acc401794 completed successfully.
> {code}
> *+6) Kill cathy1 and cathy2+*
> {code}
> root@cathy3:~/cass-0.8/bin# ./nodetool -h cathy3 ring
> Address         DC          Rack        Status State   Load            Owns    Token                                       
>                                                                                113427455640312821154458202477256070484     
> 50.57.114.45    datacenter1 rack1       Down   Normal  105.46 KB       33.33%  0                                           
> 50.57.107.176   datacenter1 rack1       Down   Normal  106 KB          33.33%  56713727820156410577229101238628035242      
> 50.57.114.39    datacenter1 rack1       Up     Normal  331.33 KB       33.33%  113427455640312821154458202477256070484 
> {code}
> *+7) Log into cassandra-cli on node3+*
> {code}
>  ./cassandra-cli -h cathy3
> Connected to: "Test Cluster" on cathy3/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] consistencylevel as ONE;
> Consistency level is set to 'ONE'.
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *+8) Bring up nodes 1 and 2 and query+*
> *NODE1 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy1
> Connected to: "Test Cluster" on cathy1/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *NODE2 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy2
> Connected to: "Test Cluster" on cathy2/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 2 Rows Returned.
> {code}
> *NODE3 - cassandra-cli*
> {code}
> ./cassandra-cli -h cathy3
> Connected to: "Test Cluster" on cathy3/9160
> Welcome to the Cassandra CLI.
> Type 'help;' or '?' for help.
> Type 'quit;' or 'exit;' to quit.
> [default@unknown] use testKS;
> Authenticated to keyspace: testKS
> [default@testKS] list metadataCF;
> Using default limit of 100
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 1 Row Returned.
> {code}
> *+9) Force Read Repair on node1+*
> {code}
> consistencylevel as THREE;
> Consistency level is set to 'THREE'.
> [default@testKS] list metadataCF;          
> Using default limit of 100
> -------------------
> RowKey: key1
> => (column=ascii_col, value=this is data inserted into ascii column, timestamp=1311035260656000)
> => (column=byte_col, value=10101010, timestamp=1311035260662000)
> => (column=int_col, value=1000, timestamp=1311035260669000)
> => (column=long_col, value=44444444444, timestamp=1311035260674000)
> => (column=utf8_col, value=this is data inserted into UTF8 column, timestamp=1311035260678000)
> => (column=uuid_col, value=e9811e90-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260666000)
> -------------------
> RowKey: key2
> => (column=ascii_col, value=key2: this is data inserted into ascii column, timestamp=1311035260682000)
> => (column=byte_col, value=0201010102, timestamp=1311035260685000)
> => (column=int_col, value=2000, timestamp=1311035260692000)
> => (column=long_col, value=222222222, timestamp=1311035260695000)
> => (column=utf8_col, value=key2-this is data inserted into UTF8 column, timestamp=1311035260699000)
> => (column=uuid_col, value=e984a100-b19d-11e0-0000-2069cd105fbf, timestamp=1311035260689000)
> 2 Rows Returned.
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message