From user-return-38990-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Feb 14 09:04:46 2014 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AEBE010DC4 for ; Fri, 14 Feb 2014 09:04:46 +0000 (UTC) Received: (qmail 37237 invoked by uid 500); 14 Feb 2014 09:04:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 36896 invoked by uid 500); 14 Feb 2014 09:04:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 36887 invoked by uid 99); 14 Feb 2014 09:04:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Feb 2014 09:04:40 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of abarua@247-inc.com designates 213.199.154.82 as permitted sender) Received: from [213.199.154.82] (HELO emea01-db3-obe.outbound.protection.outlook.com) (213.199.154.82) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Feb 2014 09:04:32 +0000 Received: from HKXPR03MB341.apcprd03.prod.outlook.com (10.141.133.12) by HKXPR03MB373.apcprd03.prod.outlook.com (10.141.133.143) with Microsoft SMTP Server (TLS) id 15.0.878.16; Fri, 14 Feb 2014 09:04:05 +0000 Received: from HKXPR03MB343.apcprd03.prod.outlook.com (10.141.133.14) by HKXPR03MB341.apcprd03.prod.outlook.com (10.141.133.12) with Microsoft SMTP Server (TLS) id 15.0.878.16; Fri, 14 Feb 2014 09:04:01 +0000 Received: from HKXPR03MB343.apcprd03.prod.outlook.com ([10.141.133.14]) by HKXPR03MB343.apcprd03.prod.outlook.com ([10.141.133.14]) with mapi id 15.00.0878.008; Fri, 14 Feb 2014 09:04:01 +0000 From: Arindam Barua To: "user@cassandra.apache.org" Subject: Bootstrap stuck: vnode enabled 1.2.12 Thread-Topic: Bootstrap stuck: vnode enabled 1.2.12 Thread-Index: Ac8pY7M6TBK7AEIyQue/LLVwiENa4Q== Date: Fri, 14 Feb 2014 09:04:00 +0000 Message-ID: <9da66398b0f347f184cb09f10c96c4d8@HKXPR03MB343.apcprd03.prod.outlook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [50.156.126.211] x-forefront-prvs: 01221E3973 x-forefront-antispam-report: SFV:NSPM;SFS:(10019001)(6039001)(6009001)(164054003)(199002)(189002)(5423002)(81686001)(74366001)(19300405004)(81816001)(90146001)(50986001)(51856001)(56816005)(33646001)(95666001)(85852003)(81342001)(74316001)(83072002)(47446002)(77982001)(87266001)(31966008)(59766001)(19580395003)(69226001)(81542001)(74662001)(2656002)(54356001)(76796001)(74876001)(87936001)(76786001)(19609705001)(83322001)(47736001)(94316002)(93516002)(65816001)(92566001)(54316002)(56776001)(93136001)(47976001)(79102001)(63696002)(80976001)(76576001)(76176001)(85306002)(53806001)(95416001)(15202345003)(15975445006)(74706001)(16236675002)(94946001)(46102001)(49866001)(86362001)(66066001)(4396001)(24736002);DIR:OUT;SFP:1102;SCL:1;SRVR:HKXPR03MB341;H:HKXPR03MB343.apcprd03.prod.outlook.com;CLIP:50.156.126.211;FPR:AC3BF207.2EE253D8.7DD31D7E.4FD7D3A9.203DD;PTR:InfoNoRecords;A:1;MX:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_9da66398b0f347f184cb09f10c96c4d8HKXPR03MB343apcprd03pro_" MIME-Version: 1.0 X-OriginatorOrg: 247-inc.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_9da66398b0f347f184cb09f10c96c4d8HKXPR03MB343apcprd03pro_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable After our otherwise successful upgrade procedure to enable vnodes, when add= ing back "new" hosts to our cluster, one non-seed host ran into a hardware = issue during bootstrap. By the time the hardware issue was fixed a week lat= er, all other nodes were added successfully, cleaned, repaired. The disks o= n this node were untouched, and when the node was started back up, it detec= ted an interrupted bootstrap, and attempted to bootstrap. However, after ~2= 4 hrs it was still stuck in the 'JOINING' state according to nodetool netst= ats on that node, even though no streams were flowing to/from it. Also, it = did not appear in nodetool status in any way/form (not even as JOINING). >From couple of observed thread dumps, the stack of the thread blocked durin= g bootstrap is at [1]. Since the node wasn't making any progress, I ended up stopping Cassandra, c= leaning up the data and commitlog directories, and attempted a fresh bootst= rap. Nodetool netstats immediately reported a whole bunch of streams queued= up, and data started streaming to the node. The data directory quickly gre= w to 18 GB (the other nodes had ~25GB, but we have lot of data with low TTL= s). However, the node ended up being in the earlier reported state, i.e. no= detool netstats doesn't have anything queued, but still reports the JOINING= state, even though it's been > 24 hrs. There are no other ERRORS in the lo= gs, and new data being written to the cluster makes it to this node just fi= ne, triggering compactions, etc from time to time. Any help is appreciated. Thanks, Arindam [1] Thread dump Thread 3708: (state =3D BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=3D0 (Compiled frame; informatio= n may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=3D14, line=3D156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterr= upt() @bci=3D1, line=3D811 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInt= erruptibly(int) @bci=3D55, line=3D969 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInter= ruptibly(int) @bci=3D24, line=3D1281 (Interpreted frame) - java.util.concurrent.CountDownLatch.await() @bci=3D5, line=3D207 (Interp= reted frame) - org.apache.cassandra.dht.RangeStreamer.fetch() @bci=3D209, line=3D256 (Interpreted frame) - org.apache.cassandra.dht.BootStrapper.bootstrap() @bci=3D120, line=3D84 (Interpreted frame) - org.apache.cassandra.service.StorageService.bootstrap(java.util.Collecti= on) @bci=3D172, line=3D978 (Interpreted frame) - org.apache.cassandra.service.StorageService.joinTokenRing(int) @bci=3D82= 7, line=3D744 (Interpreted frame) - org.apache.cassandra.service.StorageService.initServer(int) @bci=3D363, line=3D585 (Interpreted frame) - org.apache.cassandra.service.StorageService.initServer() @bci=3D4, line= =3D482 (Interpreted frame) - org.apache.cassandra.service.CassandraDaemon.setup() @bci=3D1069, line= =3D348 (Interpreted frame) - org.apache.cassandra.service.CassandraDaemon.activate() @bci=3D59, line= =3D447 (Interpreted frame) - org.apache.cassandra.service.CassandraDaemon.main(java.lang.String[]) @b= ci=3D3, line=3D490 (Interpreted frame) --_000_9da66398b0f347f184cb09f10c96c4d8HKXPR03MB343apcprd03pro_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

After our otherwise successful upgrade procedure to = enable vnodes, when adding back “new” hosts to our cluster, one= non-seed host ran into a hardware issue during bootstrap. By the time the = hardware issue was fixed a week later, all other nodes were added successfully, cleaned, repaired. The disks on this node w= ere untouched, and when the node was started back up, it detected an interr= upted bootstrap, and attempted to bootstrap. However, after ~24 hrs it was = still stuck in the ‘JOINING’ state according to nodetool netstats on that node, even though no streams were f= lowing to/from it. Also, it did not appear in nodetool status in any way/fo= rm (not even as JOINING).

 

From couple of observed thread dumps, the stack of t= he thread blocked during bootstrap is at [1].

 

Since the node wasn’t making any progress, I e= nded up stopping Cassandra, cleaning up the data and commitlog directories,= and attempted a fresh bootstrap. Nodetool netstats immediately reported a whole bunch of streams queued up, and data started streaming to= the node. The data directory quickly grew to 18 GB (the other nodes had ~2= 5GB, but we have lot of data with low TTLs). However, the node ended up bei= ng in the earlier reported state, i.e. nodetool netstats doesn’t have anything queued, but still repor= ts the JOINING state, even though it’s been > 24 hrs. There are no= other ERRORS in the logs, and new data being written to the cluster makes = it to this node just fine, triggering compactions, etc from time to time.

 

Any help is appreciated.

 

Thanks,

Arindam

[1] Thread dump
Thread 3708: (state =3D BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=3D0 (Compiled frame; i= nformation may
   be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @= bci=3D14,
   line=3D156 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCh= eckInterrupt()
   @bci=3D1, line=3D811 (Interpreted frame)
 -
   java.util.concurrent.locks.AbstractQueuedSynchronizer.doAc= quireSharedInterruptibly(int)
   @bci=3D55, line=3D969 (Interpreted frame)
 -
   java.util.concurrent.locks.AbstractQueuedSynchronizer.acqu= ireSharedInterruptibly(int)
   @bci=3D24, line=3D1281 (Interpreted frame)
 - java.util.concurrent.CountDownLatch.await() @bci=3D5, line=3D20= 7 (Interpreted
   frame)
 - org.apache.cassandra.dht.RangeStreamer.fetch() @bci=3D209, line= =3D256
   (Interpreted frame)
 - org.apache.cassandra.dht.BootStrapper.bootstrap() @bci=3D120, l= ine=3D84
   (Interpreted frame)
 - org.apache.cassandra.service.StorageService.bootstrap(java.util= .Collection)
   @bci=3D172, line=3D978 (Interpreted frame)
 - org.apache.cassandra.service.StorageService.joinTokenRing(int) = @bci=3D827,
   line=3D744 (Interpreted frame)
 - org.apache.cassandra.service.StorageService.initServer(int) @bc= i=3D363,
   line=3D585 (Interpreted frame)
 - org.apache.cassandra.service.StorageService.initServer() @bci= =3D4, line=3D482
   (Interpreted frame)
 - org.apache.cassandra.service.CassandraDaemon.setup() @bci=3D106= 9, line=3D348
   (Interpreted frame)
 - org.apache.cassandra.service.CassandraDaemon.activate() @bci=3D= 59, line=3D447
   (Interpreted frame)
 - org.apache.cassandra.service.CassandraDaemon.main(java.lang.Str= ing[]) @bci=3D3,
   line=3D490 (Interpreted frame)

--_000_9da66398b0f347f184cb09f10c96c4d8HKXPR03MB343apcprd03pro_--