Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 65465 invoked from network); 26 May 2010 09:55:26 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 May 2010 09:55:26 -0000 Received: (qmail 81187 invoked by uid 500); 26 May 2010 09:55:25 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81094 invoked by uid 500); 26 May 2010 09:55:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81086 invoked by uid 99); 26 May 2010 09:55:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 09:55:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rantav@gmail.com designates 74.125.83.44 as permitted sender) Received: from [74.125.83.44] (HELO mail-gw0-f44.google.com) (74.125.83.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 09:55:16 +0000 Received: by gwj15 with SMTP id 15so1923708gwj.31 for ; Wed, 26 May 2010 02:54:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:from:date :message-id:subject:to:content-type; bh=2fN1Clv1TtgsbtRf/H5++EVziHOpc2E9poAROBlxhpM=; b=nDGo9XkQ0728q3SEC3B3DygydW6PJqRjTwfTV8CGk9G8shYTgDUIpq55GZeTQVDNT1 wdqWtwSSBdNioYZTUbGvgmj0RqkNQE3Ru2J4mHLf5ADoCJ4huab+DSXYoYar90TnfQXJ cUL0O7j8s45QNl4qmKjGQ6bqjhyVR8BqUwSdc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; b=vI81o9iEXNwJr8HK91RQR2sRVIQgD91ZoAFhb6J+ponG2Dgn4UVeIbnSpK0E1tDsFT C5Kxjejs4VXIyqabd8bws9cSyQkjCrPKKiaKSPuVLGqeX9uVTXmHJ/CXzHKAWoJwcPxC goO1rfU/fnUfHpXlYF0U2lvJiKJP6JHXd0OF4= Received: by 10.231.155.3 with SMTP id q3mr7044846ibw.20.1274867695122; Wed, 26 May 2010 02:54:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.172.79 with HTTP; Wed, 26 May 2010 02:54:35 -0700 (PDT) From: Ran Tavory Date: Wed, 26 May 2010 12:54:35 +0300 Message-ID: Subject: nodetool move looks stuck To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00504501423a1a799a04877c440d X-Virus-Checked: Checked by ClamAV on apache.org --00504501423a1a799a04877c440d Content-Type: text/plain; charset=UTF-8 I ran nodetool move on one of the nodes and it seems stuck for a few hours now. I've been able to run it successfully in the past, but this time it looks stuck. Streams shows as if there's work in progress, but the same files have been at the same position for a few hours. I've also checked the compaction manager through jmx and it's not compacting anything. Anything I can do to continue the move operation? Is this a bug? thanks $ nodetool -h 192.168.254.58 -p 9004 streams Mode: Normal Nothing streaming to /192.168.252.124 Streaming to: /192.168.254.57 /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Filter.db 0/45587965 /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Data.db 0/18369406636 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.db 0/6163411 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db 0/85645 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.db 0/37985032 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db 0/77307418 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db 0/1436005 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db 0/704736670 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db 0/230280 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db 0/20605 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db 0/202808000 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db 0/23912 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db 0/20605 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db 0/21434973 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db 0/965028 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db 0/20605 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db 0/865380205 Not receiving any streams. $ nodetool -h 192.168.254.57 -p 9004 streams Mode: Normal Nothing streaming to /192.168.254.58 Streaming from: /192.168.254.58 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db 0/18369406636 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db 0/6163411 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db 0/85645 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db 0/37985032 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Index.db 0/230280 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Filter.db 0/20605 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db 0/202808000 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Index.db 0/23912 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Filter.db 0/20605 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db 0/21434973 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Index.db 0/965028 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Filter.db 0/20605 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db 0/865380205 --00504501423a1a799a04877c440d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I ran nodetool move on one of the nodes and it seems = stuck for a few hours now.

I've been able to r= un it successfully in the past, but this time it looks stuck.
Streams shows as if there's work in progress, but the same f= iles have been at the same position for a few hours.
I've als= o checked the compaction manager through jmx and it's not compacting an= ything.

Anything I can do to continue the move operation? Is th= is a bug?
thanks


$ n= odetool -h 192.168.254.58 -p 9004 streams
Mode: Normal
=C2=A0Nothing streaming to /192.168.252.= 124
Streaming to: /192.168.= 254.57
=C2=A0=C2=A0 /outbrain/cassandra/data/outbrain_kvdb/st= ream/KvImpressions-170-Filter.db 0/45587965
=C2=A0=C2=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressio= ns-170-Data.db 0/18369406636
=C2=A0=C2=A0 /outbrain/cassandra/dat= a/outbrain_kvdb/stream/KvRatings-142-Index.db 0/6163411
=C2=A0=C2= =A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db 0= /85645
=C2=A0=C2=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-1= 42-Data.db 0/37985032
=C2=A0=C2=A0 /outbrain/cassandra/data/outbr= ain_kvdb/stream/KvAds-108-Index.db 0/77307418
=C2=A0=C2=A0 /outbr= ain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db 0/1436005
=C2=A0=C2=A0 /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-D= ata.db 0/704736670
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace= 1/stream/Standard1-40-Index.db 0/230280
=C2=A0=C2=A0 /outbrain/ca= ssandra/data/Keyspace1/stream/Standard1-40-Filter.db 0/20605
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Da= ta.db 0/202808000
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace1= /stream/Standard1-41-Index.db 0/23912
=C2=A0=C2=A0 /outbrain/cass= andra/data/Keyspace1/stream/Standard1-41-Filter.db 0/20605
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Da= ta.db 0/21434973
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace1/= stream/Standard1-42-Index.db 0/965028
=C2=A0=C2=A0 /outbrain/cass= andra/data/Keyspace1/stream/Standard1-42-Filter.db 0/20605
=C2=A0=C2=A0 /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Da= ta.db 0/865380205
Not receiving any streams.

=
$ nodetool -h 192.168.254.57 -p 9004 streams
Mode: Normal
=C2=A0Nothing streaming to /192.168.254.5= 8
Streaming from: /192.168.= 254.58
=C2=A0=C2=A0 outbrain_kvdb: /outbrain/cassandra/data/o= utbrain_kvdb/KvImpressions-tmp-1081-Data.db 0/18369406636
=C2=A0=C2=A0 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvR= atings-tmp-167-Index.db 0/6163411
=C2=A0=C2=A0 outbrain_kvdb: /ou= tbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db 0/85645
=C2=A0=C2=A0 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/K= vRatings-tmp-167-Data.db 0/37985032
=C2=A0=C2=A0 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvA= ds-tmp-123-Index.db 0/77307418
=C2=A0=C2=A0 outbrain_kvdb: /outbr= ain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005
=C2=A0=C2=A0 outbrain_kvdb: /outbrain/cassandra/data/outbrain_kvdb/KvAds-= tmp-123-Data.db 0/704736670
=C2=A0=C2=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-t= mp-21-Index.db 0/230280
=C2=A0=C2=A0 Keyspace1: /outbrain/cassand= ra/data/Keyspace1/Standard1-tmp-21-Filter.db 0/20605
=C2=A0=C2=A0= Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db 0/2= 02808000
=C2=A0=C2=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-t= mp-22-Index.db 0/23912
=C2=A0=C2=A0 Keyspace1: /outbrain/cassandr= a/data/Keyspace1/Standard1-tmp-22-Filter.db 0/20605
=C2=A0=C2=A0 = Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db 0/21= 434973
=C2=A0=C2=A0 Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-t= mp-23-Index.db 0/965028
=C2=A0=C2=A0 Keyspace1: /outbrain/cassand= ra/data/Keyspace1/Standard1-tmp-23-Filter.db 0/20605
=C2=A0=C2=A0= Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db 0/8= 65380205



--00504501423a1a799a04877c440d--