Return-Path: 
 <user-return-26623-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91C669FC5
	for <apmail-cassandra-user-archive@www.apache.org>;
 Mon,  4 Jun 2012 03:15:39 +0000 (UTC)
Received: (qmail 74167 invoked by uid 500); 4 Jun 2012 03:15:36 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 74052 invoked by uid 500); 4 Jun 2012 03:15:35 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 74025 invoked by uid 99); 4 Jun 2012 03:15:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jun 2012 03:15:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of
 gcdcu-cassandra-user-1@m.gmane.org designates 80.91.229.3 as permitted
 sender)
Received: from [80.91.229.3] (HELO plane.gmane.org) (80.91.229.3)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jun 2012 03:15:28 +0000
Received: from list by plane.gmane.org with local (Exim 4.69)
	(envelope-from <gcdcu-cassandra-user-1@m.gmane.org>)
	id 1SbNl1-0000jT-TF
	for user@cassandra.apache.org; Mon, 04 Jun 2012 05:15:04 +0200
Received: from 122-116-61-83.HINET-IP.hinet.net ([122.116.61.83])
        by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
        id 1AlnuQ-0007hv-00
        for <user@cassandra.apache.org>; Mon, 04 Jun 2012 05:15:03 +0200
Received: from koji.lin by 122-116-61-83.HINET-IP.hinet.net with local (Gmexim
 0.1 (Debian))
        id 1AlnuQ-0007hv-00
        for <user@cassandra.apache.org>; Mon, 04 Jun 2012 05:15:03 +0200
X-Injected-Via-Gmane: http://gmane.org/
To: user@cassandra.apache.org
From: koji <koji.lin@gmail.com>
Subject: Re: Node join streaming stuck at 100%
Date: Mon, 4 Jun 2012 03:12:29 +0000 (UTC)
Lines: 82
Message-ID: <loom.20120604T050547-127@post.gmane.org>
References: 
 <376CEC01195C894CB9F8A3C274029A96BD0731DE@fish-ex2k10-03.azaleos.net>
 <62BAD014-6065-48CA-9ED6-F82DF245505F@thelastpickle.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Complaints-To: usenet@dough.gmane.org
X-Gmane-NNTP-Posting-Host: sea.gmane.org
User-Agent: Loom/3.14 (http://gmane.org/)
X-Loom-IP: 122.116.61.83 (Mozilla/5.0 (Macintosh;
 Intel Mac OS X 10_7_3) AppleWebKit/536.5 (KHTML,
 like Gecko) Chrome/19.0.1084.53 Safari/536.5)
X-Virus-Checked: Checked by ClamAV on apache.org


aaron morton <aaron <at> thelastpickle.com> writes:

> 
> Did you restart ? All good?
> Cheers
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
>  <at> aaronmorton
> http://www.thelastpickle.com
> 
> 
> On 27/04/2012, at 9:49 AM, Bryce Godfrey wrote:
> 
> This is the second node I’ve joined to my cluster in the last few days, and 
so far both have become stuck at 100% on a large file according to netstats.  
This is on 1.0.9, is there anything I can do to make it move on besides 
restarting Cassandra?  I don’t see any errors or warns in logs for 
either server, and there is plenty of disk space.
> 
>  
> On the sender side I see this:
> 
> Streaming to: /10.20.1.152
> 
>    /opt/cassandra/data/MonitoringData/PropertyTimeline-hc-80540-Data.db 
sections=1 progress=82393861085/82393861085 - 100%
> 
>  
> On the node joining I don’t see this file in netstats, and all pending 
streams are sitting at 0%
> 
>  
>  


Hi
we have the same problem (1.0.7) , our netstats log is like this:

Mode: NORMAL
Streaming to: /1.1.1.1
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3757-Data.db 
   sections=1234 progress=3256666/3256666 - 100%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3641-Data.db 
   sections=4386 progress=0/1025272214 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3761-Data.db 
   sections=2956 progress=0/17826723 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3730-Data.db 
   sections=3792 progress=0/56066299 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3760-Data.db 
   sections=4384 progress=0/90941161 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3687-Data.db 
   sections=3958 progress=0/54729557 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OfflineMessage-hc-3762-Data.db 
   sections=766 progress=0/2605165 - 0%
Streaming to: /1.1.1.2
   /mnt/ebs1/cassandra-data/data/NemoModel/OneWayFriend-hc-709-Data.db 
   sections=3228 progress=29175698/29175698 - 100%
   /mnt/ebs1/cassandra-data/data/NemoModel/OneWayFriend-hc-789-Data.db 
   sections=2102 progress=0/618938 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OneWayFriend-hc-765-Data.db 
   sections=3044 progress=0/1996687 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OneWayFriend-hc-788-Data.db 
   sections=2773 progress=0/1374636 - 0%
   /mnt/ebs1/cassandra-data/data/NemoModel/OneWayFriend-hc-729-Data.db 
   sections=3150 progress=0/22111512 - 0%
 Nothing streaming from /1.1.1.1
 Nothing streaming from /1.1.1.2
Pool Name                    Active   Pending      Completed
Commands                        n/a         1       23825242
Responses                       n/a        25       19644808


After restart, the pending streams are cleared, but next time we do 
"nodetool repair -pr" again, the pending still happened. And this always 
happend on same node(we have total 12 nodes).

koji