From user-return-5436-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu May 06 14:45:39 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 56854 invoked from network); 6 May 2010 14:45:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 May 2010 14:45:39 -0000 Received: (qmail 36190 invoked by uid 500); 6 May 2010 14:45:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 36094 invoked by uid 500); 6 May 2010 14:45:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 36086 invoked by uid 99); 6 May 2010 14:45:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 May 2010 14:45:38 +0000 X-ASF-Spam-Status: No, hits=-0.2 required=10.0 tests=AWL,RCVD_IN_DNSWL_NONE,SPF_PASS,T_HK_NAME_DR X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [195.190.135.10] (HELO mx.expurgate.net) (195.190.135.10) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 May 2010 14:45:32 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: Strange streaming behaviour Date: Thu, 6 May 2010 16:45:09 +0200 In-Reply-To: Thread-Topic: Strange streaming behaviour Thread-Index: Acrq2WNmLB7Yq4o5Tc+FZFjl4c9OdQCUNPCA References: <1O8q7E-00034H-Tq@mail.eleven.de> From: =?iso-8859-1?Q?Dr=2E_Martin_Grabm=FCller?= To: , Message-ID: <1OA2K6-0007jE-Br@mail.eleven.de> Gary, thanks for the feedback. Today, I tried to reproduce, but have not been able to. I successfully bootstrapped two new nodes into my cluster while the cluster was under load, but streaming did not get stuck this time. Maybe somehow my local setup is responsible or my original diagnostics was not correct. I will create a ticket when I have been able to reproduce. Unfortunately, running with debug logging enabled is too slow to load gigabytes of data into a cluster in a sensible time frame. Martin=20 > -----Original Message----- > From: Gary Dusbabek [mailto:gdusbabek@gmail.com]=20 > Sent: Monday, May 03, 2010 5:57 PM > To: user@cassandra.apache.org > Subject: Re: Strange streaming behaviour >=20 > Martin, >=20 > Please create a ticket and include the relevant parts of your=20 > storage-conf. >=20 > To summarize, the output gives you the impression that bootstrap has > completed normally, but when you check, it appears to be hung on the > receiving nodes? >=20 > Do you mind turning debug on and and seeing if you can reproduce? The > strange part is that the source node doesn't think it's streaming at > all. >=20 > Gary. >=20 >=20 > On Mon, May 3, 2010 at 02:30, Dr. Martin Grabm=FCller > wrote: > > Hello list, > > > > I encountered a problem with streaming in my test cluster but > > found nothing like this in JIRA or on the list. > > > > I'm running a test cluster of three nodes, RF=3D3, Cassandra 0.6.1. > > I started the first node and inserted some data, then bootstrapped > > the other machines one after the other. =A0No problems here. > > > > Then, I inserted data over the weekend and checked back=20 > this morning, > > finding a lot of -tmp-files in the data directory of the=20 > two bootstrapepd > > machines which seem to be leftovers from streaming. > > > > Interesting thing is, they are mostly all about the same=20 > size on each node. Additionally, > > this size is the same as the size of some file to be=20 > streamed (see the output of > > nodetool streams at the end of this post): > > > > [Node 2:] > > > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A02360 2010-05-03 07:39=20 > AccountList-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A02360 2010-05-03 00:39=20 > AddressList-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A02360 2010-05-03 07:53=20 > IndexQueue-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A02360 2010-05-03 05:19=20 > MailMetadata-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A02360 2010-05-03 00:53=20 > Statistics-tmp-7-Data.db > > > > [Node 3:] > > > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 10994 2010-05-03 07:39=20 > AccountList-tmp-6-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 =A0 325 2010-05-03 00:39=20 > AddressList-tmp-6-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 10994 2010-05-03 08:02=20 > CustomerList-tmp-6-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 10994 2010-05-03 07:53=20 > IndexQueue-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 10994 2010-05-03 01:02=20 > MailMetadata-tmp-7-Data.db > > -rw-r--r-- 1 cassandra cassandra =A0 =A0 10994 2010-05-03 00:53=20 > Statistics-tmp-7-Data.db > > > > Checking the logs, I find a few streaming-related error messages: > > > > [Node 1 (yes, the node without leftover files):] > > > > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675=20 > DebuggableThreadPoolExecutor.java (line 101) Error in=20 > ThreadPoolExecutor > > java.lang.RuntimeException: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2- > Data.db (No such file or directory) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:34) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > ecutor.java:1110) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE > xecutor.java:603) > > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2- > Data.db (No such file or directory) > > =A0 =A0 =A0 =A0at java.io.RandomAccessFile.open(Native Method) > > =A0 =A0 =A0 =A0at = java.io.RandomAccessFile.(RandomAccessFile.java:233) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream > Task.java:63) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:30) > > =A0 =A0 =A0 =A0... 3 more > > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675=20 > CassandraDaemon.java (line 78) Fatal exception in thread=20 > Thread[MESSAGE-STREAMING-POOL:1,5,main] > > java.lang.RuntimeException: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2- > Data.db (No such file or directory) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:34) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > ecutor.java:1110) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE > xecutor.java:603) > > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2- > Data.db (No such file or directory) > > =A0 =A0 =A0 =A0at java.io.RandomAccessFile.open(Native Method) > > =A0 =A0 =A0 =A0at = java.io.RandomAccessFile.(RandomAccessFile.java:233) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream > Task.java:63) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:30) > > =A0 =A0 =A0 =A0... 3 more > > > > [Node 3:] > > > > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656=20 > DebuggableThreadPoolExecutor.java (line 101) Error in=20 > ThreadPoolExecutor > > java.lang.RuntimeException: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db=20 > (No such file or directory) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:34) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > ecutor.java:1110) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE > xecutor.java:603) > > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db=20 > (No such file or directory) > > =A0 =A0 =A0 =A0at java.io.RandomAccessFile.open(Native Method) > > =A0 =A0 =A0 =A0at = java.io.RandomAccessFile.(RandomAccessFile.java:233) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream > Task.java:63) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:30) > > =A0 =A0 =A0 =A0... 3 more > > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-03 00:39:46,656=20 > CassandraDaemon.java (line 78) Fatal exception in thread=20 > Thread[MESSAGE-STREAMING-POOL:1,5,main] > > java.lang.RuntimeException: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db=20 > (No such file or directory) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:34) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > ecutor.java:1110) > > =A0 =A0 =A0 =A0at=20 > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE > xecutor.java:603) > > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException:=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-7-Data.db=20 > (No such file or directory) > > =A0 =A0 =A0 =A0at java.io.RandomAccessFile.open(Native Method) > > =A0 =A0 =A0 =A0at = java.io.RandomAccessFile.(RandomAccessFile.java:233) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStream > Task.java:63) > > =A0 =A0 =A0 =A0at=20 > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable > .java:30) > > =A0 =A0 =A0 =A0... 3 more > > > > On Node 2, there are no errors. > > > > I have not been able to reproduce this error with the=20 > default configuration. > > > > Has anybody an idea how to debug this problem? =A0Is this=20 > known? =A0Should I create > > a ticket? > > > > Thanks in advance! > > > > Martin > > > > P.S. Here is the output of nodetool streams for the three nodes: > > > > [Node 1:] 192.168.211.44 > > > > Mode: Normal > > =A0Nothing streaming to /192.168.211.46 > > =A0Nothing streaming to /192.168.211.45 > > Not receiving any streams. > > > > [Node 2:] 192.168.211.45 > > > > Mode: Normal > > Streaming to: /192.168.211.46 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-6-Data. > db 10994/10994 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index. > db 0/3142 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter > .db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.d > b 0/15974 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Inde > x.db 0/758284 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filt > er.db 0/16525 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data > .db 0/5581529 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Index > .db 0/592 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-6-Data. > db 0/2866 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter > .db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.d > b 0/8801484 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Inde > x.db 0/61 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-6-Data > .db 0/229 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db=20 > 0/9148876 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-88-Index.d > b 0/59273 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-88-Filter. > db 0/36685 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-88-Data.db > 0/109059024 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db > 0/632998 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.d > b 0/24685 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db=20 > 0/5148176 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Inde > x.db 0/2360 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data > .db 0/10994 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Index > .db 0/3142 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-12-Data. > db 0/15996 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Ind > ex.db 0/789717 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Fil > ter.db 0/29005 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Dat > a.db 0/5791834 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Inde > x.db 0/592 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-12-Data > .db 0/2866 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index > .db 0/648 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data. > db 0/5280403 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Ind > ex.db 0/61 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Fil > ter.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-12-Dat > a.db 0/229 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.d > b 0/12171 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db > 0/14228834 > > =A0Nothing streaming to /192.168.211.44 > > Streaming from: /192.168.211.46 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-7-Data.db 0/229 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9050963 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Index.db 0/58383 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Filter.db 0/36685 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-87-Data.db=20 > 0/107327613 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/646210 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24925 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5254628 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Index.db 0/3142 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-13-Data.db 0/15995 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index. > db 0/802581 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter > .db 0/29125 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.d > b 0/5885615 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Index.db 0/592 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-13-Data.db 0/2866 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db=20 > 0/5310585 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Index.db 0/61 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Filter > .db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-13-Data.db 0/229 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14129376 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Index.db=20 > 2360/3142 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db > 2360/5673882 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-7-Data.db=20 > 2360/2866 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter. > db 2360/16645 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db=20 > 2360/8830815 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-7-Data.db=20 > 2360/10994 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db=20 > 2360/15971 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.d > b 2360/770984 > > > > [Node 3:] 192.168.211.46 > > > > Mode: Normal > > =A0Nothing streaming to /192.168.211.44 > > Streaming to: /192.168.211.45 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Index > .db 2360/2360 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-7-Data. > db 0/10994 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Index. > db 0/3142 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Filter > .db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-6-Data.d > b 0/15971 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Inde > x.db 0/770984 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Filt > er.db 0/16645 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-6-Data > .db 0/5673882 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Index > .db 0/592 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-7-Data. > db 0/2866 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Index.db 0/648 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Filter > .db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-6-Data.d > b 0/8830815 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Inde > x.db 0/61 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-7-Data > .db 0/229 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Index.db 0/12171 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Filter.db 0/445 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-6-Data.db=20 > 0/9050963 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-86-Index.d > b 0/58383 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-86-Filter. > db 0/36685 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailList-86-Data.db > 0/107327613 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Index.db > 0/646210 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Filter.d > b 0/24925 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Headers-11-Data.db=20 > 0/5254628 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Inde > x.db 0/2360 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AddressList-12-Data > .db 0/10994 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Index > .db 0/3142 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/Statistics-13-Data. > db 0/15995 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Ind > ex.db 0/802581 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Fil > ter.db 0/29125 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/MailMetadata-12-Dat > a.db 0/5885615 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Inde > x.db 0/592 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Filt > er.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/AccountList-13-Data > .db 0/2866 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Index > .db 0/648 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Filte > r.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/IndexQueue-12-Data. > db 0/5310585 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Ind > ex.db 0/61 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Fil > ter.db 0/325 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/CustomerList-13-Dat > a.db 0/229 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Index.d > b 0/12171 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Filter.db 0/445 > > =A0=20 > /mnt/data000/cassandra/data/Archive/stream/EventLog-12-Data.db > 0/14129376 > > Streaming from: /192.168.211.45 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Index.db 0/12171 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Filter.db 0/445 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-7-Data.db 0/9148876 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Index.db 0/59273 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Filter.db 0/36685 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailList-tmp-88-Data.db=20 > 0/109059024 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Index.db 0/632998 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Filter.db 0/24685 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Headers-tmp-12-Data.db 0/5148176 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Index.db 0/2360 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-13-Data.db 0/10994 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Index.db 0/3142 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-12-Data.db 0/15996 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Index. > db 0/789717 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Filter > .db 0/29005 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-13-Data.d > b 0/5791834 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Index.db 0/592 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AccountList-tmp-12-Data.db 0/2866 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Index.db 0/648 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Filter.db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-13-Data.db=20 > 0/5280403 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Index.db 0/61 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Filter > .db 0/325 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/CustomerList-tmp-12-Data.db 0/229 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Index.db 0/12171 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Filter.db 0/445 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/EventLog-tmp-13-Data.db 0/14228834 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Index.d > b 10994/758284 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/AddressList-tmp-6-Data.db=20 > 325/10994 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/IndexQueue-tmp-7-Data.db=20 > 10994/8801484 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Data.db > 10994/5581529 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/MailMetadata-tmp-7-Filter. > db 10994/16525 > > =A0 Archive:=20 > /mnt/data000/cassandra/data/Archive/Statistics-tmp-7-Data.db=20 > 10994/15974 > > >=20