Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 97583 invoked from network); 26 May 2010 18:55:35 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 May 2010 18:55:35 -0000 Received: (qmail 36375 invoked by uid 500); 26 May 2010 18:55:34 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 36356 invoked by uid 500); 26 May 2010 18:55:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 36348 invoked by uid 99); 26 May 2010 18:55:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 18:55:34 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=10.0 tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-wy0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 18:55:28 +0000 Received: by wye20 with SMTP id 20so2584818wye.31 for ; Wed, 26 May 2010 11:55:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=FoFF+1RC/ItVF/4hseE96IPeCnuMjMLfLNyIHqBogCI=; b=Pb0TGd/AMMyL1fdMMAalAUc24pfdKep502c9H9KrJEoDrph3k2qS0FBVHb8Pcr1cM7 17XWO5j5CIcXFdeTQfQfY6q6cknpQrtHziu4J3h7RmhhYr05RDNUZfeP+LIsMCo3aC7K DkJ46cQs5drwfCga7kvwedn8FGFD4Ki7tdC4k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=VScolSOdUCMnuLNpgke781rSf8GQ/LYBMoBx6cYJU7d2l6/5MiShjEzG+xZ7JMYl1Z Vd2+ElWkHrXss3Xu9NX1tFhscSBvOouAYrot2jwcxhteuv3oN142whYUHjrQtxiRB9nM dLYo8p5S6Uok85EVhQee5WNTuzgC6tpV/vwqU= Received: by 10.227.134.197 with SMTP id k5mr8704945wbt.95.1274900107405; Wed, 26 May 2010 11:55:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.17.197 with HTTP; Wed, 26 May 2010 11:54:47 -0700 (PDT) In-Reply-To: References: From: Jonathan Ellis Date: Wed, 26 May 2010 12:54:47 -0600 Message-ID: Subject: Re: nodetool move looks stuck To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Are there any exceptions in the log like the one in https://issues.apache.org/jira/browse/CASSANDRA-1019 ? If so you'll need to restart the moving node and try again. On Wed, May 26, 2010 at 3:54 AM, Ran Tavory wrote: > I ran nodetool move on one of the nodes and it seems stuck for a few hour= s > now. > I've been able to run it successfully in the past, but this time it looks > stuck. > Streams shows as if there's work in progress, but the same files have bee= n > at the same position for a few hours. > I've also checked the compaction manager through jmx and it's not compact= ing > anything. > Anything I can do to continue the move operation? Is this a bug? > thanks > > $ nodetool -h 192.168.254.58 -p 9004 streams > Mode: Normal > =A0Nothing streaming to /192.168.252.124 > Streaming to: /192.168.254.57 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Fi= lter.db > 0/45587965 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Da= ta.db > 0/18369406636 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.= db > 0/6163411 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter= .db > 0/85645 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.d= b > 0/37985032 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db > 0/77307418 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db > 0/1436005 > =A0=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db > 0/704736670 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db 0/= 230280 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db 0= /20605 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db > 0/202808000 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db 0/= 23912 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db 0= /20605 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db 0/2= 1434973 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db 0/= 965028 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db 0= /20605 > =A0=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db > 0/865380205 > Not receiving any streams. > $ nodetool -h 192.168.254.57 -p 9004 streams > Mode: Normal > =A0Nothing streaming to /192.168.254.58 > Streaming from: /192.168.254.58 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db > 0/18369406636 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db 0/61634= 11 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db 0/8564= 5 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db 0/379850= 32 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005 > =A0=A0 outbrain_kvdb: > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Ind= ex.db > 0/230280 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Fil= ter.db > 0/20605 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Dat= a.db > 0/202808000 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Ind= ex.db > 0/23912 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Fil= ter.db > 0/20605 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Dat= a.db > 0/21434973 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Ind= ex.db > 0/965028 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Fil= ter.db > 0/20605 > =A0=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Dat= a.db > 0/865380205 > > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com