Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 11008 invoked from network); 14 Nov 2010 22:17:34 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 14 Nov 2010 22:17:34 -0000 Received: (qmail 95815 invoked by uid 500); 14 Nov 2010 22:17:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 95788 invoked by uid 500); 14 Nov 2010 22:17:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 95780 invoked by uid 99); 14 Nov 2010 22:17:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 22:17:56 +0000 X-ASF-Spam-Status: No, hits=2.7 required=10.0 tests=FREEMAIL_FROM,FS_LARGE_PERCENT2,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.160.172 as permitted sender) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 22:17:48 +0000 Received: by gyh20 with SMTP id 20so2720998gyh.31 for ; Sun, 14 Nov 2010 14:17:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=nL3fXs0Mru5/nBwikN/akeTVF9mzWxRuEV1N8fF2dOk=; b=ul06iY89KaLiAHCNwujoP915KejeBY9NLgJ0XPthQEQX2tjqqOTT7Mn7oDUh2h+h1B I5o9xmQyvf1CDCWrYUvKXuRC/jjz10lJAMfXCwq95W7jDRpL/671uruQXgBZGj6rEIjj xJZat/Qtou9Dsfjn+UP3wUwu/QqIj9tGjbvBc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=oixqOoCEGrGqsFySGevAvTC6SRelC0WT6sddH5LHeAWD6VXFk/VnG4DFbxYOnpsqi2 j1PZ4mwwtcqNE0e8I5JnLroKazRH9Y+LbtBzuLODsBMCjKcGNFGBUwCMpcPEC7EPZNEp iCgYQlnEZdTBHheirYQ95huz/LYeTpAsYpFXk= MIME-Version: 1.0 Received: by 10.151.47.14 with SMTP id z14mr8239684ybj.314.1289773045557; Sun, 14 Nov 2010 14:17:25 -0800 (PST) Received: by 10.236.109.169 with HTTP; Sun, 14 Nov 2010 14:17:25 -0800 (PST) In-Reply-To: References: Date: Sun, 14 Nov 2010 16:17:25 -0600 Message-ID: Subject: Re: repair takes two days, and ends up stuck: stream at 1096% (yes, really) From: Jonathan Ellis To: user Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org What exception is causing it to fail/retry? On Sun, Nov 14, 2010 at 3:49 PM, Chip Salzenberg wrote= : > My by-now infamous eight-node cluster running 0.7.0beta3+ dropped many > replication MUTATEs during load, so I decided to fix replication copies w= ith > a "nodetool repair" on one of the nodes (X.21). =A0The repair has been ru= nning > for two days, and has finally gotten itself wedged into a state where it > can't proceed. > The log on X.21 continually describe the need to stream a data file, > unsuccessfully. =A0From other clues below I gather this is a receive stre= am. > This message repeated many many times, multiple per second, but has now > stopped: > INFO [Thread-13877] 2010-11-14 09:17:35,207 StreamInSession.java (line 12= 4) > Streaming of file > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197079) > =A0=A0 =A0 =A0 =A0 progress=3D90112/219682197079 - 0% from > org.apache.cassandra.streaming.StreamInSession@3ee3da2c failed: requestin= g a > retry. > Here's the best joke, though: "nodetool -h X.20 nestats" shows that the > given stream has been attempted a few times and is still being attempted, > but in a broken way, such that the progress percentage has gone way past > 100%. =A0It's now at 1096% and still rising. > I'm not rebooting so I can poke around as devs suggest. =A0I'm also not > sending logs to the list, at least in part because they're, well, big. = =A0If > any developers want them, though, I'm happy to send them. > -------------------------------------------------------------------------= ------------------------------------------ > Mode: Normal > Streaming to: /X.21 > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197= 079) > =A0=A0 =A0 =A0 =A0 progress=3D2408587638800/219682197079 - 1096% > =A0 =A0 =A0 =A0 <---- see this > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-386-Data.db/(0,182528797= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/182528797 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,908075169= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/908075169 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-382-Data.db/(0,784362565= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/784362565 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,896956312= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/896956312 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,894019840= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/894019840 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-380-Data.db/(0,901377643= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/901377643 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-333-Data.db/(0,22306924) > =A0=A0 =A0 =A0 =A0 progress=3D0/22306924 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,888814566= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888814566 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-374-Data.db/(0,889095219= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/889095219 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,893034298= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893034298 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-389-Data.db/(0,371718620= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/371718620 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-319-Data.db/(0,141728308= 70) > =A0=A0 =A0 =A0 =A0 progress=3D0/14172830870 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-283-Data.db/(0,893940731= 6) > =A0=A0 =A0 =A0 =A0 progress=3D0/8939407316 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897417147= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/897417147 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,357220526= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/357220526 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,899103394= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/899103394 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,898165901= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/898165901 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,133239573= 68) > =A0=A0 =A0 =A0 =A0 progress=3D0/13323957368 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,892116147= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/892116147 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-383-Data.db/(0,282162393= 03) > =A0=A0 =A0 =A0 =A0 progress=3D0/28216239303 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-330-Data.db/(0,307921317= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/307921317 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,185277927= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/185277927 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,893683568= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893683568 - 0% > Streaming from: /X.21 > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-440-Data.db/(0,176842211) > =A0=A0 =A0 =A0 =A0 progress=3D0/176842211 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,447272883) > =A0=A0 =A0 =A0 =A0 progress=3D0/447272883 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-412-Data.db/(0,444440243) > =A0=A0 =A0 =A0 =A0 progress=3D0/444440243 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,14275850800) > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-397-Data.db/(0,31878407176) > =A0=A0 =A0 =A0 =A0 progress=3D0/31878407176 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-393-Data.db/(0,446800028) > =A0=A0 =A0 =A0 =A0 progress=3D0/446800028 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-439-Data.db/(0,367116560) > =A0=A0 =A0 =A0 =A0 progress=3D0/367116560 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,445241132) > =A0=A0 =A0 =A0 =A0 progress=3D0/445241132 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,4497953871) > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-396-Data.db/(0,449662908) > =A0=A0 =A0 =A0 =A0 progress=3D0/449662908 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-409-Data.db/(0,454101872) > =A0=A0 =A0 =A0 =A0 progress=3D0/454101872 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,447381444) > =A0=A0 =A0 =A0 =A0 progress=3D0/447381444 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633237) > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 = =A0 =A0Completed > Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0709910236 > Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 =A0= 0 =A0 =A0 =A0363174385 > -------------------------------------------------------------------------= ------------------------------------------ > > Meanwhile, "nodetool -h X.21 netstats" shows a large number of transfers > that are at 0% and haven't moved, AFAICT, for at least an hour: > -------------------------------------------------------------------------= ------------------------------------------ > Mode: Normal > Streaming to: /X.20 > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633237= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-439-Data.db/(0,367116560= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/367116560 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,449795387= 1) > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-412-Data.db/(0,444440243= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/444440243 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-440-Data.db/(0,176842211= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/176842211 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,142758508= 00) > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,445241132= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/445241132 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-397-Data.db/(0,318784071= 76) > =A0=A0 =A0 =A0 =A0 progress=3D0/31878407176 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,447272883= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/447272883 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-393-Data.db/(0,446800028= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/446800028 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-396-Data.db/(0,449662908= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/449662908 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,447381444= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/447381444 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-409-Data.db/(0,454101872= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/454101872 - 0% > Streaming to: /X.22 > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-350-Data.db/(0,887780227= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/887780227 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-341-Data.db/(0,885896138= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/885896138 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-368-Data.db/(0,892560053= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/892560053 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-358-Data.db/(0,888436251= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888436251 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,893446845= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893446845 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-354-Data.db/(0,889058842= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/889058842 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-90-Data.db/(0,6150503130= 1) > =A0=A0 =A0 =A0 =A0 progress=3D0/61505031301 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,887620464= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/887620464 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-361-Data.db/(0,890820399= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/890820399 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-345-Data.db/(0,887535512= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/887535512 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-329-Data.db/(0,168761073= 70) > =A0=A0 =A0 =A0 =A0 progress=3D0/16876107370 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-364-Data.db/(0,893839028= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893839028 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-356-Data.db/(0,891862436= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/891862436 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,886276363= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/886276363 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-292-Data.db/(0,388239771= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/388239771 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-379-Data.db/(0,907731463= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/907731463 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-348-Data.db/(0,893114355= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893114355 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-371-Data.db/(0,888682755= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888682755 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-338-Data.db/(0,885144435= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/885144435 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-340-Data.db/(0,890937418= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/890937418 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-233-Data.db/(0,339025560= 16) > =A0=A0 =A0 =A0 =A0 progress=3D0/33902556016 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897426603= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/897426603 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,888711957= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888711957 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-327-Data.db/(0,208633237= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/208633237 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,893954909= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893954909 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,897265056= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/897265056 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-339-Data.db/(0,888998653= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888998653 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,901053427= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/901053427 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-271-Data.db/(0,449795387= 1) > =A0=A0 =A0 =A0 =A0 progress=3D0/4497953871 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-343-Data.db/(0,891732427= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/891732427 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-357-Data.db/(0,888267065= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888267065 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-335-Data.db/(0,889998928= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/889998928 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-365-Data.db/(0,888528931= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/888528931 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-328-Data.db/(0,142758508= 00) > =A0=A0 =A0 =A0 =A0 progress=3D0/14275850800 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-355-Data.db/(0,893535664= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/893535664 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-347-Data.db/(0,891375566= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/891375566 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,897994571= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/897994571 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,897589898= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/897589898 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-336-Data.db/(0,891079134= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/891079134 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,892852094= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/892852094 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-337-Data.db/(0,885983148= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/885983148 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-346-Data.db/(0,886424157= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/886424157 - 0% > =A0=A0 /var/lib/cassandra/data/Attrs/TestAttrs-e-353-Data.db/(0,889222127= ) > =A0=A0 =A0 =A0 =A0 progress=3D0/889222127 - 0% > Streaming from: /X.20 > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-385-Data.db/(0,357220526) > =A0=A0 =A0 =A0 =A0 progress=3D0/357220526 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-389-Data.db/(0,371718620) > =A0=A0 =A0 =A0 =A0 progress=3D0/371718620 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-369-Data.db/(0,888814566) > =A0=A0 =A0 =A0 =A0 progress=3D0/888814566 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-374-Data.db/(0,889095219) > =A0=A0 =A0 =A0 =A0 progress=3D0/889095219 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-330-Data.db/(0,307921317) > =A0=A0 =A0 =A0 =A0 progress=3D0/307921317 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-331-Data.db/(0,13323957368) > =A0=A0 =A0 =A0 =A0 progress=3D0/13323957368 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-383-Data.db/(0,28216239303) > =A0=A0 =A0 =A0 =A0 progress=3D0/28216239303 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-319-Data.db/(0,14172830870) > =A0=A0 =A0 =A0 =A0 progress=3D0/14172830870 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-366-Data.db/(0,899103394) > =A0=A0 =A0 =A0 =A0 progress=3D0/899103394 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-367-Data.db/(0,894019840) > =A0=A0 =A0 =A0 =A0 progress=3D0/894019840 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-390-Data.db/(0,185277927) > =A0=A0 =A0 =A0 =A0 progress=3D0/185277927 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-378-Data.db/(0,896956312) > =A0=A0 =A0 =A0 =A0 progress=3D0/896956312 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-372-Data.db/(0,893683568) > =A0=A0 =A0 =A0 =A0 progress=3D0/893683568 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-386-Data.db/(0,182528797) > =A0=A0 =A0 =A0 =A0 progress=3D0/182528797 - 0% > =A0=A0 Attrs: /var/lib/cassandra/data/Attrs/TestAttrs-e-333-Data.db/(0,22= 306924) > =A0=A0 =A0 =A0 =A0 progress=3D0/22306924 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-376-Data.db/(0,897417147) > =A0=A0 =A0 =A0 =A0 progress=3D0/897417147 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-283-Data.db/(0,8939407316) > =A0=A0 =A0 =A0 =A0 progress=3D0/8939407316 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-381-Data.db/(0,908075169) > =A0=A0 =A0 =A0 =A0 progress=3D0/908075169 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-373-Data.db/(0,892116147) > =A0=A0 =A0 =A0 =A0 progress=3D0/892116147 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-377-Data.db/(0,898165901) > =A0=A0 =A0 =A0 =A0 progress=3D0/898165901 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-332-Data.db/(0,219682197079) > =A0=A0 =A0 =A0 =A0 progress=3D0/219682197079 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-382-Data.db/(0,784362565) > =A0=A0 =A0 =A0 =A0 progress=3D0/784362565 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-375-Data.db/(0,893034298) > =A0=A0 =A0 =A0 =A0 progress=3D0/893034298 - 0% > =A0=A0 Attrs: > /var/lib/cassandra/data/Attrs/TestAttrs-e-380-Data.db/(0,901377643) > =A0=A0 =A0 =A0 =A0 progress=3D0/901377643 - 0% > =A0Nothing streaming from /10.5.5.22 > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 = =A0 =A0Completed > Commands =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0n/a =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0433633667 > Responses =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 n/a =A0 =A0 =A0 =A0= 0 =A0 =A0 =A0402612386 > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com