Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 95134 invoked from network); 15 Nov 2010 18:30:32 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 15 Nov 2010 18:30:32 -0000 Received: (qmail 6888 invoked by uid 500); 15 Nov 2010 18:31:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 6869 invoked by uid 500); 15 Nov 2010 18:31:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 6861 invoked by uid 99); 15 Nov 2010 18:31:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Nov 2010 18:31:01 +0000 X-ASF-Spam-Status: No, hits=2.7 required=10.0 tests=FREEMAIL_FROM,FS_LARGE_PERCENT2,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Nov 2010 18:30:53 +0000 Received: by yxd39 with SMTP id 39so62581yxd.31 for ; Mon, 15 Nov 2010 10:30:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=SwRssLVYlvUpu4/veNTNrVFwHHOYh3RCNNKUDZOa8is=; b=b2KEYdtuWox4GT+9L934BKRLmFDZEb5etA+jjyruUaEYjHJ6tbslNMmCrz+OHp1okY 2ZHyY+VgLBzjRyA36n2ao3VbcfcG1c/QQ+KGFWj6/kJ2grEcPaoc/eKzX6DbCjCQEn8W DBikWqAB+MDVv1jF8kicpHcmD/aI9ChgiWvn0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=Ozh8PtG9rcV8qrSgaKdGCj25opt0iryLwx4bsZfPy5DxPOOYA26zCpfDhfjqU83em/ gv7qQWdVkfEZu5tSFpEm/8xOWAJGKvFypkkEkGU8p2pf91u3/wHzp/eiaS7dBa4O0gXw 9unqWCiHmtT6rPLaK2jKBBbwx/MCYPZJFCJkk= MIME-Version: 1.0 Received: by 10.151.47.14 with SMTP id z14mr9939783ybj.314.1289845830959; Mon, 15 Nov 2010 10:30:30 -0800 (PST) Received: by 10.236.109.169 with HTTP; Mon, 15 Nov 2010 10:30:30 -0800 (PST) In-Reply-To: References: Date: Mon, 15 Nov 2010 12:30:30 -0600 Message-ID: Subject: Re: repair takes two days, and ends up stuck: stream at 1096% (yes, really) From: Jonathan Ellis To: user Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Is X.20 spewing these errors constantly now? Did X.21 log anything when/before the errors started on X.20? It looks to me like you had an out-of-Cassandra's-control network problem (or maybe X.21 restarted) and the retry code isn't working. On Sun, Nov 14, 2010 at 7:26 PM, Chip Salzenberg wrote= : > Excerpts from the log of X.20... =A0Hope you can make something of it. > -------------------------------------------------------------------------= ------------------------------ > =A0INFO [MiscStage:1] 2010-11-14 17:19:33,105 StreamReplyVerbHandler.java > (line 58) Need to re-stream file > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db to /X.21 > ERROR [Streaming:284690] 2010-11-14 17:19:33,105 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.RuntimeException: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo0(Native Method= ) > =A0=A0 =A0 =A0 =A0at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:456) > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImp= l.java:557) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:= 102) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.= java:64) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > =A0=A0 =A0 =A0 =A0... 3 more > ERROR [Streaming:284690] 2010-11-14 17:19:33,106 > AbstractCassandraDaemon.java (line 90) Fatal exception in thread > Thread[Streaming:284690,1,main] > java.lang.RuntimeException: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo0(Native Method= ) > =A0=A0 =A0 =A0 =A0at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:456) > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImp= l.java:557) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:= 102) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.= java:64) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > =A0=A0 =A0 =A0 =A0... 3 more > ERROR [Streaming:284691] 2010-11-14 17:19:33,107 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.RuntimeException: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.IOException: Broken pipe > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo0(Native Method= ) > =A0=A0 =A0 =A0 =A0at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:456) > =A0=A0 =A0 =A0 =A0at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImp= l.java:557) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:= 102) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.= java:64) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > =A0=A0 =A0 =A0 =A0... 3 more > =A0INFO [MiscStage:1] 2010-11-14 17:19:33,171 StreamReplyVerbHandler.java > (line 58) Need to re-stream file > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db to /X.21 > ... > -------------------------------------------------------------------------= ------------------------------ > On Sun, Nov 14, 2010 at 2:17 PM, Jonathan Ellis wrote= : >> >> What exception is causing it to fail/retry? >> >> On Sun, Nov 14, 2010 at 3:49 PM, Chip Salzenberg >> wrote: >> > My by-now infamous eight-node cluster running 0.7.0beta3+ dropped many >> > replication MUTATEs during load, so I decided to fix replication copie= s >> > with >> > a "nodetool repair" on one of the nodes (X.21). =A0The repair has been >> > running >> > for two days, and has finally gotten itself wedged into a state where = it >> > can't proceed. >> > The log on X.21 continually describe the need to stream a data file, >> > unsuccessfully. =A0From other clues below I gather this is a receive >> > stream. >> > This message repeated many many times, multiple per second, but has no= w >> > stopped: >> > INFO [Thread-13877] 2010-11-14 09:17:35,207 StreamInSession.java (line >> > 124) >> > Streaming of file >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197079) >> > =A0=A0 =A0 =A0 =A0 progress=3D90112/219682197079 - 0% from >> > org.apache.cassandra.streaming.StreamInSession@3ee3da2c failed: >> > requesting a >> > retry. >> > Here's the best joke, though: "nodetool -h X.20 nestats" shows that th= e >> > given stream has been attempted a few times and is still being >> > attempted, >> > but in a broken way, such that the progress percentage has gone way pa= st >> > 100%. =A0It's now at 1096% and still rising. >> > I'm not rebooting so I can poke around as devs suggest. =A0I'm also no= t >> > sending logs to the list, at least in part because they're, well, big. >> > =A0If >> > any developers want them, though, I'm happy to send them. >> > >> > ----------------------------------------------------------------------= --------------------------------------------- >> > Mode: Normal >> > Streaming to: /X.21 >> > >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197079) >> > =A0=A0 =A0 =A0 =A0 progress=3D2408587638800/219682197079 - 1096% >> > =A0 =A0 =A0 =A0 <---- see this >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-386-Data.db/(0,182528= 797) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/182528797 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,908075= 169) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/908075169 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-382-Data.db/(0,784362= 565) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/784362565 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,896956= 312) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/896956312 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,894019= 840) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/894019840 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-380-Data.db/(0,901377= 643) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/901377643 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-333-Data.db/(0,223069= 24) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/22306924 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,888814= 566) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888814566 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-374-Data.db/(0,889095= 219) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/889095219 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,893034= 298) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893034298 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-389-Data.db/(0,371718= 620) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/371718620 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-319-Data.db/(0,141728= 30870) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/14172830870 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-283-Data.db/(0,893940= 7316) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/8939407316 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897417= 147) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897417147 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,357220= 526) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/357220526 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,899103= 394) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/899103394 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,898165= 901) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/898165901 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,133239= 57368) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/13323957368 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,892116= 147) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/892116147 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-383-Data.db/(0,282162= 39303) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/28216239303 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-330-Data.db/(0,307921= 317) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/307921317 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,185277= 927) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/185277927 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,893683= 568) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893683568 - 0% >> > Streaming from: /X.21 >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-440-Data.db/(0,176842211) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/176842211 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,447272883) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/447272883 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-412-Data.db/(0,444440243) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/444440243 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,14275850800) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-397-Data.db/(0,31878407176) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/31878407176 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-393-Data.db/(0,446800028) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/446800028 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-439-Data.db/(0,367116560) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/367116560 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,445241132) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/445241132 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,4497953871) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-396-Data.db/(0,449662908) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/449662908 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-409-Data.db/(0,454101872) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/454101872 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,447381444) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/447381444 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633237) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% >> > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending = =A0 =A0 =A0Completed >> > Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 = =A0 =A0 0 =A0 =A0 =A0709910236 >> > Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0363174385 >> > >> > ----------------------------------------------------------------------= --------------------------------------------- >> > >> > Meanwhile, "nodetool -h X.21 netstats" shows a large number of transfe= rs >> > that are at 0% and haven't moved, AFAICT, for at least an hour: >> > >> > ----------------------------------------------------------------------= --------------------------------------------- >> > Mode: Normal >> > Streaming to: /X.20 >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633= 237) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-439-Data.db/(0,367116= 560) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/367116560 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,449795= 3871) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-412-Data.db/(0,444440= 243) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/444440243 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-440-Data.db/(0,176842= 211) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/176842211 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,142758= 50800) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,445241= 132) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/445241132 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-397-Data.db/(0,318784= 07176) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/31878407176 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,447272= 883) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/447272883 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-393-Data.db/(0,446800= 028) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/446800028 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-396-Data.db/(0,449662= 908) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/449662908 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,447381= 444) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/447381444 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-409-Data.db/(0,454101= 872) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/454101872 - 0% >> > Streaming to: /X.22 >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-350-Data.db/(0,887780= 227) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/887780227 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-341-Data.db/(0,885896= 138) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/885896138 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-368-Data.db/(0,892560= 053) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/892560053 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-358-Data.db/(0,888436= 251) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888436251 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,893446= 845) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893446845 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-354-Data.db/(0,889058= 842) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/889058842 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-90-Data.db/(0,6150503= 1301) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/61505031301 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,887620= 464) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/887620464 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-361-Data.db/(0,890820= 399) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/890820399 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-345-Data.db/(0,887535= 512) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/887535512 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-329-Data.db/(0,168761= 07370) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/16876107370 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-364-Data.db/(0,893839= 028) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893839028 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-356-Data.db/(0,891862= 436) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/891862436 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,886276= 363) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/886276363 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-292-Data.db/(0,388239= 771) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/388239771 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-379-Data.db/(0,907731= 463) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/907731463 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-348-Data.db/(0,893114= 355) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893114355 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-371-Data.db/(0,888682= 755) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888682755 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-338-Data.db/(0,885144= 435) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/885144435 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-340-Data.db/(0,890937= 418) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/890937418 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-233-Data.db/(0,339025= 56016) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/33902556016 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897426= 603) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897426603 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,888711= 957) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888711957 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633= 237) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,893954= 909) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893954909 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,897265= 056) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897265056 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-339-Data.db/(0,888998= 653) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888998653 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,901053= 427) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/901053427 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,449795= 3871) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-343-Data.db/(0,891732= 427) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/891732427 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-357-Data.db/(0,888267= 065) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888267065 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-335-Data.db/(0,889998= 928) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/889998928 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-365-Data.db/(0,888528= 931) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888528931 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,142758= 50800) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-355-Data.db/(0,893535= 664) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893535664 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-347-Data.db/(0,891375= 566) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/891375566 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,897994= 571) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897994571 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,897589= 898) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897589898 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-336-Data.db/(0,891079= 134) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/891079134 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,892852= 094) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/892852094 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-337-Data.db/(0,885983= 148) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/885983148 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-346-Data.db/(0,886424= 157) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/886424157 - 0% >> > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-353-Data.db/(0,889222= 127) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/889222127 - 0% >> > Streaming from: /X.20 >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,357220526) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/357220526 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-389-Data.db/(0,371718620) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/371718620 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,888814566) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/888814566 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-374-Data.db/(0,889095219) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/889095219 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-330-Data.db/(0,307921317) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/307921317 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,13323957368) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/13323957368 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-383-Data.db/(0,28216239303) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/28216239303 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-319-Data.db/(0,14172830870) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/14172830870 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,899103394) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/899103394 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,894019840) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/894019840 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,185277927) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/185277927 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,896956312) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/896956312 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,893683568) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893683568 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-386-Data.db/(0,182528797) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/182528797 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-333-Data.db/(0,22306924) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/22306924 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897417147) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/897417147 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-283-Data.db/(0,8939407316) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/8939407316 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,908075169) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/908075169 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,892116147) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/892116147 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,898165901) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/898165901 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197079) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/219682197079 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-382-Data.db/(0,784362565) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/784362565 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,893034298) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/893034298 - 0% >> > =A0=A0 Attrs: >> > /var/lib/cassandra/data/Attrs/TestAttrs-e-380-Data.db/(0,901377643) >> > =A0=A0 =A0 =A0 =A0 progress=3D0/901377643 - 0% >> > =A0Nothing streaming from /10.5.5.22 >> > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending = =A0 =A0 =A0Completed >> > Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 = =A0 =A0 0 =A0 =A0 =A0433633667 >> > Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0402612386 >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com