incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haithem Jarraya <haithem.jarr...@struq.com>
Subject Re: Repair Hanging C* 1.2.4
Date Thu, 02 May 2013 13:13:22 GMT
Hi Yuki,

I think I found what went wrong, one box in WDC had the disk full during
scrub. The box become unusable but kept alive status.
It was not supposed to go down or marked as dead?
By the way can we specify the snpashot output directory in nodetool
command, so that does not happen again?

Thanks,

H


On 2 May 2013 13:17, Yuki Morishita <mor.yuki@gmail.com> wrote:

> Hi,
>
>
> ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line
> 2420) Repair session failed:****
>
> java.lang.IllegalArgumentException: Requested range intersects a local
> range but is not fully contained in one; this would lead to imprecise repair
>
>
> This error means you are repairing the range that spreads across multiple
> (virtual) nodes.
> I think this won't happen unless you specify the repair range with -st and
> -et option.
>
> How do you start repair?
>
> --
> Yuki Morishita
> Sent with Airmail <http://airmailapp.info/tracking>
>
> On May 2, 2013 at May 2, 2013, Haithem Jarraya (haithem.jarraya@struq.com)
> wrote:
>
> Hi All, ****
>
> ** **
>
> Cassandra repair has been a real pain for us and it’s holding back our
> migration from mongo for quiet sometimes now.****
>
> We saw errors like this during the repair, ****
>
>  INFO [AntiEntropyStage:1] 2013-05-01 14:30:54,300 AntiEntropyService.java
> (line 764) [repair #ed104480-b26a-11e2-af9b-05179fa66b76] mycolumnfamily is
> fully synced (1 remaining column family to sync for this session)****
>
> ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line
> 2420) Repair session failed:****
>
> java.lang.IllegalArgumentException: Requested range intersects a local
> range but is not fully contained in one; this would lead to imprecise repair
> ****
>
>         at
> org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:175)
> ****
>
>         at
> org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:621)
> ****
>
>         at
> org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:610)
> ****
>
>         at
> org.apache.cassandra.service.AntiEntropyService.submitRepairSession(AntiEntropyService.java:127)
> ****
>
>         at
> org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:2480)
> ****
>
>         at
> org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2416)
> ****
>
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)***
> *
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)***
> *
>
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)****
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)****
>
>         at java.lang.Thread.run(Thread.java:662)****
>
> ** **
>
> ** **
>
> Ok we might have gone beyond the GCGrades(again before repair do not
> complete) So we ran scrub  in all node in parallel as it was suggested in
> this mailing list.****
>
> I am not sure if this can be the cause of the problem or not, but in
> reality we had this issue of repair not completing and hanging from the day
> we started testing cassandra 1.2.2, same issue happening with every upgrade
> 1.2.3 and now 1.2.4.****
>
> I want a way to kick in the repair  if they hang or cancel the previous
> one without restarting the cluster, we can’t afford to do that the day we
> go live.****
>
> ** **
>
> Let me start by presenting our current configuration.****
>
> Data Centers:****
>
> 2 Data center (Amsterdam 6 nodes with RF of 3, Washington D.C with RF of 1)
> ****
>
> 1 Key space with 3 column families ~= 100GB of data.****
>
> Each node running Cassandra 1.2.4 with Java6_update45 running centos 2.6
> with 32GB of RAM, 24 Cores @2.00GHZ, JNA v3.2.4 installed, 2 disk( 1
> rotational for os and commit logs, and 1 ssd for the data). We are getting
> really good read performances, 99% < 10ms,  95% < 5ms.****
>
> ** **
>
> nodetool status ****
>
> Datacenter: ams01****
>
> =================****
>
> Status=Up/Down****
>
> |/ State=Normal/Leaving/Joining/Moving****
>
> --  Address       Load       Tokens  Owns   Host
> ID                               Rack****
>
> UN  x.x.x.23   34.04 GB   256     13.1%
> 4a7bc489-25af-4c20-80f8-499ffcb18e2d  RAC1****
>
> UN  x.x.x.79    28.53 GB   256     12.6%
> 98a1167f-cf75-4201-a454-695e0f7d2d72  RAC1****
>
> UN  x.x.x.78    41.31 GB   256     11.9%
> 62a418b5-3c38-4f66-874d-8138d6d565e5  RAC1****
>
> UN  x.x.x.66   54.41 GB   256     13.8%
> ab564d16-4081-4866-b8ba-26461d9a93d7  RAC1****
>
> UN  x.x.x.91    45.92 GB   256     12.6%
> 2e1e7179-82e6-4ae6-b986-383acc9fc8a2  RAC1****
>
> UN  x.x.x.126  37.31 GB   256     11.8%
> d4bed3b1-ffaf-4c68-b560-d270355c8c4b  RAC1****
>
> Datacenter: wdc01****
>
> =================****
>
> Status=Up/Down****
>
> |/ State=Normal/Leaving/Joining/Moving****
>
> --  Address       Load       Tokens  Owns   Host
> ID                               Rack****
>
> UN  x.x.x.144   30.64 GB   256     12.0%
> 1860011e-fa7c-4ce1-ad6b-c8a38a5ddd02  RAC1****
>
> UN  x.x.x.140   86.05 GB   256     12.3%
> f3fa985d-5056-4ddc-b146-d02432c3a86e  RAC1****
>
> ** **
>
> nodetool  status mykeyspace****
>
> Datacenter: ams01****
>
> =================****
>
> Status=Up/Down****
>
> |/ State=Normal/Leaving/Joining/Moving****
>
> --  Address       Load       Tokens  Owns (effective)  Host
> ID                               Rack****
>
> UN  x.x.x.66   54.41 GB   256     53.6%
> ab564d16-4081-4866-b8ba-26461d9a93d7  RAC1****
>
> UN  x.x.x.91    45.92 GB   256     52.1%
> 2e1e7179-82e6-4ae6-b986-383acc9fc8a2  RAC1****
>
> UN  x.x.x.126  37.31 GB   256     47.9%
> d4bed3b1-ffaf-4c68-b560-d270355c8c4b  RAC1****
>
> UN  x.x.x.23   34.04 GB   256     50.9%
> 4a7bc489-25af-4c20-80f8-499ffcb18e2d  RAC1****
>
> UN  x.x.x.79    28.53 GB   256     47.4%
> 98a1167f-cf75-4201-a454-695e0f7d2d72  RAC1****
>
> UN  x.x.x.78    41.31 GB   256     48.0%
> 62a418b5-3c38-4f66-874d-8138d6d565e5  RAC1****
>
> Datacenter: wdc01****
>
> =================****
>
> Status=Up/Down****
>
> |/ State=Normal/Leaving/Joining/Moving****
>
> --  Address       Load       Tokens  Owns (effective)  Host
> ID                               Rack****
>
> UN  x.x.x140   86.05 GB   256     51.5%
> f3fa985d-5056-4ddc-b146-d02432c3a86e  RAC1****
>
> UN  x.x.x.144   30.64 GB   256     48.5%
>    1860011e-fa7c-4ce1-ad6b-c8a38a5ddd02  RAC1****
>
> ** **
>
> The first thing we notice is that the data distribution is off by few %,
> well I guess, if we had a repair running with no hanging this should fix
> the problem.****
>
> Now when we run a repair it usually hangs after a day, there is no more
> repair messages in the logs or stream, cpu usage goes down.****
>
> This the output for nodetool tpstats and nodetool netstats in all boxes.**
> **
>
> ** **
>
> Amsterdam data center:****
>
> NODE 1:****
>
> x.x.x.23 (I run the first repair on this node), we can see the
> AntiEntropySession 4 Active and 7 pending****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         1       33134051
> 0                 0****
>
> RequestResponseStage              0         0       60655547
> 0                 0****
>
> MutationStage                     0         0       46000521
> 0                 0****
>
> ReadRepairStage                   0         0        2712610
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         848258
> 0                 0****
>
> AntiEntropyStage                  0         0          24530         0
>                0****
>
> MigrationStage                    0         0             56
> 0                 0****
>
> MemtablePostFlusher               0         0           3295
> 0                 0****
>
> FlushWriter                       0         0           1487
> 0                 0****
>
> MiscStage                         0         0            245
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               4         7            891
> 0                 0****
>
> InternalResponseStage             0         0              7
> 0                 0****
>
> HintedHandoff                     0         0              7
>   0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> For nodetool stats,  it says nothing streaming from x.x.x.140 which is the
> ip of 1 nodes in WDC datacentre.****
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140 ****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       60710552****
>
> Responses                       n/a         0       56269681****
>
> ** **
>
> NODE 2:****
>
> x.x.x.126 nothing going on this node.****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0       34896655
> 0                 0****
>
> RequestResponseStage              0         0       74815324
> 0                 0****
>
> MutationStage                     0         0       44827842
> 0                 0****
>
> ReadRepairStage                   0         0        3175404
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         971226
>   0                 0****
>
> AntiEntropyStage                  0         0          11904
> 0                 0****
>
> MigrationStage                    0         0             59
> 0                 0****
>
> MemtablePostFlusher               0         0           7029
> 0                 0****
>
> FlushWriter                       0         0           3000
> 0                 0****
>
> MiscStage                         0         0           1300
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0              7
> 0                 0****
>
> HintedHandoff                     0         0              8
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> Same output for netstats as the previous node****
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       75174907****
>
> Responses                       n/a         0       55239904****
>
> ** **
>
> NODE 3:****
>
> x.x.x.78 after the repair stopped in NODE 1, I tried to run a repair on
> this node to see if, it will change things or not (I think I tried to run
> it twice)****
>
> We can see 4 AntiEntropySession Active and 6 Pending****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0       29928646
> 0                 0****
>
> RequestResponseStage              0         0       81431526
> 0                 0****
>
> MutationStage                     0         0       46631197
> 0                 0****
>
> ReadRepairStage                   0         0        3352193
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         994857
> 0                 0****
>
> AntiEntropyStage                  0         0          32477
> 0                 0****
>
> MigrationStage                    0         0             44
> 0                 0****
>
> MemtablePostFlusher               0         0           7521         0
>                0****
>
> FlushWriter                       0         0           3110
> 0                 0****
>
> MiscStage                         0         0           1023
> 0                 0****
>
> commitlog_archiver                0         0              0
>         0                 0****
>
> AntiEntropySessions               4         6            360
> 0                 0****
>
> InternalResponseStage             0         0             70
> 0                 0****
>
> HintedHandoff                     0         0              9
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> ** **
>
> We can see for the netstats that it is waiting ****
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       81478294****
>
> Responses                       n/a         3       50728352****
>
> ** **
>
> NODE 4:****
>
> x.x.x. 66****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0       31542526
> 0                 0****
>
> RequestResponseStage              0         0       66173136
> 0                 0****
>
> MutationStage                     0         0       46796311
> 0                 0****
>
> ReadRepairStage                   0         0        2542891
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         726267
> 0                 0****
>
> AntiEntropyStage                  0         0           3782
>     0                 0****
>
> MigrationStage                    0         0             50
> 0                 0****
>
> MemtablePostFlusher               0         0           2807
> 0                 0****
>
> FlushWriter                       0         0           1400
> 0                 2****
>
> MiscStage                         0         0            679
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0              3
> 0                 0****
>
> HintedHandoff                     0         0              8
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       66201849****
>
> Responses                       n/a         0       54649566****
>
> ** **
>
> NODE 5****
>
> x.x.x.79****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0        4807546
> 0                 0****
>
> RequestResponseStage              0         0       15208415         0
>                 0****
>
> MutationStage                     0         0       17640854
> 0                 0****
>
> ReadRepairStage                   0         0         208035
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         241855
> 0                 0****
>
> AntiEntropyStage                  0         0           1096
> 0                 0****
>
> MigrationStage                    0         0             31
> 0                 0****
>
> MemtablePostFlusher               0         0            801
> 0                 0****
>
> FlushWriter                       0         0            351
> 0                 0****
>
> MiscStage                         0         0            101
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0             10
> 0                 0****
>
> HintedHandoff                     0         0              0
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> # nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       15216079****
>
> Responses                       n/a         2       20432998****
>
> ** **
>
> NODE 6:****
>
> x.x.x.91****
>
> nodetool tpstats****
>
> tPool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0       50507669
> 0                 0****
>
> RequestResponseStage              0         0       72430667
> 0                 0****
>
> MutationStage                     0         0       47096834
> 0                 0****
>
> ReadRepairStage                   0         0        3135286
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         728625
> 0                 0****
>
> AntiEntropyStage                  0         0           3996
> 0                 0****
>
> MigrationStage                    0         0             57
> 0                 0****
>
> MemtablePostFlusher               0         0           2941
> 0                 0****
>
> FlushWriter                       0         0           1453
> 0                 2****
>
> MiscStage                         0         0            743
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0              3
> 0                 0****
>
> HintedHandoff                     0         0             10
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Nothing streaming from /x.x.x.140****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0       73084789****
>
> Responses                       n/a         0       66121629****
>
> ** **
>
> ** **
>
> WDC datacentre:****
>
> In this node we see a different tpstats, it shows one MiscStage 1 Active
> and 28 pending, why is that?****
>
> The netstats shows that all the streaming are not progressing.****
>
> Node 1:****
>
> x.x.x.140****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0        7780233
> 0                 0****
>
> RequestResponseStage              0         0              0
> 0                 0****
>
> MutationStage                     0         0       36732109
> 0                 0****
>
> ReadRepairStage                   0         0              0
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
> 0                 0****
>
> GossipStage                       0         0         641497
> 0                 0****
>
> AntiEntropyStage                  0         0           4834
> 0                 0****
>
> MigrationStage                    0         0             65
> 0                 0****
>
> MemtablePostFlusher               1         5           4400
> 0                 0****
>
> FlushWriter                       0         0           2771
> 0                 0****
>
> MiscStage                         1        28           1720
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0             15
> 0                 0****
>
> HintedHandoff                     0         0              8
>   0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR                  0****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                     0****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Nothing streaming to /x.x.x.91****
>
> Streaming from: /x.x.x.91****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-31657-Data.db
> sections=1192 progress=0/2001599 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-33861-Data.db
> sections=8 progress=0/36381 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-33900-Data.db
> sections=2 progress=0/35827 - 0%****
>
> Streaming from: /x.x.x.23****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-20361-Data.db
> sections=1 progress=0/35973 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-19809-Data.db
> sections=5809 progress=0/7701015 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-20297-Data.db
> sections=8 progress=0/36494 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-19444-Data.db
> sections=1191 progress=0/1964863 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily3/mykeyspace-mycloumnfamily3-ib-10019-Data.db
> sections=2338 progress=0/5357560 - 0%****
>
> Streaming from: /x.x.x.78****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily3/mykeyspace-mycloumnfamily3-ib-15329-Data.db
> sections=2338 progress=0/5358677 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-31112-Data.db
> sections=1 progress=0/36877 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-28005-Data.db
> sections=4026 progress=0/7804220 - 0%****
>
> Streaming from: /x.x.x.66****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-42913-Data.db
> sections=4026 progress=0/7803966 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-39649-Data.db
> sections=1345 progress=0/184273 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-41289-Data.db
> sections=1138 progress=0/1471186 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42101-Data.db
> sections=133 progress=0/74800 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42399-Data.db
> sections=23 progress=0/36965 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42447-Data.db
> sections=3 progress=0/36404 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-41119-Data.db
> sections=2057 progress=0/1234797 - 0%****
>
>    mykeyspace:
> /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-40226-Data.db
> sections=4632 progress=0/5706080 - 0%****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0           2826****
>
> Responses                       n/a         4       40771253****
>
> ** **
>
> NODE 2:****
>
> x.x.x.140****
>
> nodetool tpstats****
>
> Pool Name                    Active   Pending      Completed   Blocked
> All time blocked****
>
> ReadStage                         0         0        3412912
> 0                 0****
>
> RequestResponseStage              0         0              4         0
>             0****
>
> MutationStage                     0         0       22540846
> 0                 0****
>
> ReadRepairStage                   0         0              0
> 0                 0****
>
> ReplicateOnWriteStage             0         0              0
>      0                 0****
>
> GossipStage                       0         0         296181
> 0                 0****
>
> AntiEntropyStage                  0         0           6276
> 0                 0****
>
> MigrationStage                    0         0             24
> 0                 0****
>
> MemtablePostFlusher               0         0           5092
> 0                 0****
>
> FlushWriter                       0         0           2953
> 0                 0****
>
> MiscStage                         0         0           1811
> 0                 0****
>
> commitlog_archiver                0         0              0
> 0                 0****
>
> AntiEntropySessions               0         0              0
> 0                 0****
>
> InternalResponseStage             0         0              8
> 0                 0****
>
> HintedHandoff                     0         0              7
> 0                 0****
>
> ** **
>
> Message type           Dropped****
>
> RANGE_SLICE                  0****
>
> READ_REPAIR               8942****
>
> BINARY                       0****
>
> READ                         0****
>
> MUTATION                114559****
>
> _TRACE                       0****
>
> REQUEST_RESPONSE             0****
>
> ** **
>
> nodetool netstats****
>
> Mode: NORMAL****
>
> Not sending any streams.****
>
> Not receiving any streams.****
>
> Pool Name                    Active   Pending      Completed****
>
> Commands                        n/a         0           1581****
>
> Responses                       n/a         0       25108161****
>
> ** **
>
> ** **
>
> ** **
>
> Many thanks,****
>
> ** **
>
> ** **
>
> Haithem****
>
>

Mime
View raw message