Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6EAEF101FE for ; Thu, 9 Jan 2014 03:24:43 +0000 (UTC) Received: (qmail 73858 invoked by uid 500); 9 Jan 2014 03:24:29 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 73750 invoked by uid 500); 9 Jan 2014 03:24:25 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 73735 invoked by uid 99); 9 Jan 2014 03:24:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 03:24:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.220.53 as permitted sender) Received: from [209.85.220.53] (HELO mail-pa0-f53.google.com) (209.85.220.53) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 03:24:17 +0000 Received: by mail-pa0-f53.google.com with SMTP id hz1so2722484pad.12 for ; Wed, 08 Jan 2014 19:23:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7WZow0akQ+pCOiPH8w246OxA3I/tLPqdkhbwjKCK7C4=; b=q6Q2fRURHjBsfgq2BWg8lQ3srIgHAnSeYSC8ch9FYsObrfv5DyJe4A1GvZLbOC27w3 cTJ/lMhQc4CP+RQpa4dsQeozWxltJDxFTiOAF59UHDBofHmfts4ifun9jH3gQHpqoNiQ Gj2hHFDMk7QYfHeskVTN+nF29iqnQPdRGT2Jz1qG5Qjsg5Q+LTfP+gRtuz4YTD23VOKo XCXt5/DVIBJPaMsHue9cSNd9uB/f8aEyOgmU5rKPpvPYuU9sPEWLj4OqtG1TZzswfGqw q41Yf6EYZBSZYozstF8vtSTfPhNhlp7GqIhOG78HxZf+6/ZtYpg6AddSChtju/bJD8Zw 8rTQ== MIME-Version: 1.0 X-Received: by 10.66.144.137 with SMTP id sm9mr946198pab.64.1389237836717; Wed, 08 Jan 2014 19:23:56 -0800 (PST) Received: by 10.70.16.226 with HTTP; Wed, 8 Jan 2014 19:23:56 -0800 (PST) In-Reply-To: <931B7EEA-7E9C-4842-AA86-2F45147E783C@163.com> References: <1C649EB0-FB3B-4B76-80DA-97BACD5BF634@163.com> <931B7EEA-7E9C-4842-AA86-2F45147E783C@163.com> Date: Wed, 8 Jan 2014 19:23:56 -0800 Message-ID: Subject: Re: graceful_stop.sh hung From: Ted Yu To: hzwangxx , "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=047d7b6d8a3ac4220904ef812368 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d8a3ac4220904ef812368 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Can you check region server log on inspur253.deu.edu.cn,60020,1388053123213 ? Cheers On Wed, Jan 8, 2014 at 5:34 PM, hzwangxx wrote: > Hi, Ted > This is the master log: > > *2014-01-08 18:40:48,640 [IPC Server handler 37 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Addeu.edu.cnded move plan > hri=3Dtest,|a1bd417af20749* > *110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751.78c953d5= 3f6498664d9a067701a7e7d7., > src=3Dinspur251.deu.edu.cn ,60020,1388050383= 789, > d* > *est=3Dinspur255.deu.edu.cn > ,60020,1388056934052, running balancer* > *2014-01-08 18:40:48,860 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|a1bd4* > *17af20749110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751= .78c953d53f6498664d9a067701a7e7d7. > that was online on inspur255.deu.edu.cn * > *,60020,1388056934052* > *2014-01-08 18:40:50,690 [IPC Server handler 44 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|a786b578f29c43* > *1374b62ca7df559277|1380621867798|7273081556605895,1382673452778.c621b3bf= 29262ca5248c03a8d6ebb41e., > src=3Dinspur251.deu.edu.cn ,60020,1388050383= 789, > d* > *est=3Dinspur253.deu.edu.cn > ,60020,1388053123213, running balancer* > *2014-01-08 18:40:51,078 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|a786b* > *578f29c431374b62ca7df559277|1380621867798|7273081556605895,1382673452778= .c621b3bf29262ca5248c03a8d6ebb41e. > that was online on inspur253.deu.edu.cn * > *,60020,1388053123213* > *2014-01-08 18:40:53,944 [IPC Server handler 46 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|bucket-ynote-o* > *nline|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098946420,13= 86199418157.4dad873a6af4d3a9809339281c3cb34c., > src=3Dinspur251.deu.edu.cn ,60* > *020,1388050383789, dest=3Dinspur254.deu.edu.cn > ,60020,1388054917364, running balancer* > *2014-01-08 18:40:55,416 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|bucke* > *t-ynote-online|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098= 946420,1386199418157.4dad873a6af4d3a9809339281c3cb34c. > that was online on ins* > *pur254.deu.edu.cn ,60020,1388054917364* > *2014-01-08 18:40:57,067 [IPC Server handler 7 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|c4617a74baabdd0* > *6f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237.0e311941f= 5ff202bcefe57aa4079a188., > src=3Dinspur251.deu.edu.cn ,60020,1388050383= 789, > de* > *st=3Dinspur253.deu.edu.cn > ,60020,1388053123213, running balancer* > *2014-01-08 18:40:57,511 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|c4617* > *a74baabdd06f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237= .0e311941f5ff202bcefe57aa4079a188. > that was online on inspur253.deu.edu.cn * > *,60020,1388053123213* > *2014-01-08 18:41:00,143 [IPC Server handler 23 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|coursera-video* > *|0dc2a6efeba02749d6187481d8f18357|1380039238151|41650362285832984,138085= 3043651.3a955559fb65caf32a05e18b8b6b93f8., > src=3Dinspur251.deu.edu.cn ,60020,* > *1388050383789, dest=3Dinspur308.deu.edu.cn > ,60020,1388059770705, running balancer* > *2014-01-08 18:41:00,989 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|cours* > *era-video|0dc2a6efeba02749d6187481d8f18357|1380039238151|416503622858329= 84,1380853043651.3a955559fb65caf32a05e18b8b6b93f8. > that was online on inspur3* > *08.deu.edu.cn ,60020,1388059770705* > *2014-01-08 18:41:01,904 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:41:02,569 [IPC Server handler 24 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|dedb0fec255422* > *f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018.34f3ef51= 6fe6fba940eeb0902b9acd3d., > src=3Dinspur251.deu.edu.cn ,60020,1388050383= 789, > d* > *est=3Dinspur254.deu.edu.cn > ,60020,1388054917364, running balancer* > *2014-01-08 18:41:02,873 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|dedb0* > *fec255422f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018= .34f3ef516fe6fba940eeb0902b9acd3d. > that was online on inspur254.deu.edu.cn * > *,60020,1388054917364* > *2014-01-08 18:41:04,184 [IPC Server handler 44 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=3Dtest,|e4e49102c1e6ea* > *97094a40c57420a628|1381085596723|42696720858381436,1382457119052.ac201a5= 6d80f13ca5357d474578a91c2., > src=3Dinspur251.deu.edu.cn > ,60020,1388050383789, * > *dest=3Dinspur308.deu.edu.cn > ,60020,1388059770705, running balancer* > *2014-01-08 18:41:04,863 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|e4e49* > *102c1e6ea97094a40c57420a628|1381085596723|42696720858381436,138245711905= 2.ac201a56d80f13ca5357d474578a91c2. > that was online on inspur308.photo.163.or* > *g,60020,1388059770705* > *2014-01-08 18:43:46,735 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:46:51,704 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:47:03,294 [935521285@qtp-711761606-0] INFO > org.apache.zookeeper.ZooKeeper - Initiating client connection, > connectString=3Dinspur254.phot* > *o.163.org :2181,inspur253.deu.edu.cn > :2181,inspur252.deu.edu.cn > :2181,inspur251.deu.edu.cn > :2181,inspur255.deu.edu.cn > :2181 sessionTimeout=3D120* > *0000 > watcher=3Dcatalogtracker-on-org.apache.hadoop.hbase.client.HConnectionMan= ager$HConnectionImplementation@172b29ed* > *2014-01-08 18:47:03,295 [935521285@qtp-711761606-0] INFO > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - The identifier = of > this process is * > *25086@inspur249.deu.edu.cn <25086@inspur249.deu.edu.cn>* > *2014-01-08 18:47:03,295 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > :2181)] INFO org.apache.zookeeper.ClientCnx= n > - Opening socket c* > *onnection to server inspur251.deu.edu.cn/172.17.7.1:2181 > . Will not attempt to > authenticate using SASL (Unable to locate a login configuration)* > *2014-01-08 18:47:03,297 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > :2181)] INFO org.apache.zookeeper.ClientCnx= n > - Socket connectio* > *n established to inspur251.deu.edu.cn/172.17.7.1:2181 > , initiating session* > *2014-01-08 18:47:03,302 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > :2181)] INFO org.apache.zookeeper.ClientCnx= n > - Session establishment complete on server > inspur251.deu.edu.cn/172.17.7.1:2181 > , sessionid =3D > 0x42b69781e0f11d, negotiated timeout =3D 300000* > *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0] INFO > org.apache.zookeeper.ZooKeeper - Session: 0x42b69781e0f11d closed* > *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0-EventThread] INFO > org.apache.zookeeper.ClientCnxn - EventThread shut down* > *2014-01-08 18:58:19,346 [IPC Reader 6 on port 60000] WARN > org.apache.hadoop.ipc.HBaseServer - Incorrect header or version mismatch > from 172.17.4.249:54719 got version 4 expecte= d > version 3* > > *I killed the process around **2014-01-08 18:55.T**he hanging region (* > 812912be704946d24c5f1b5e3184b2f5*) has not any log.* > > *Thanks* > =E5=9C=A8 2014=E5=B9=B41=E6=9C=888=E6=97=A5=EF=BC=8C23:58=EF=BC=8CTed Yu = =E5=86=99=E9=81=93=EF=BC=9A > > Can you pastebin master log around 2014-01-08 18:40 ? > > Thanks > > > On Wed, Jan 8, 2014 at 3:57 AM, hzwangxx wrote: > >> Hi, all >> I restart a region server by using graceful_stop.sh >> (bin/graceful_stop.sh --restart --reload --debug hostname), when running= a >> moment, the process hanging as follows: >> >> 2014-01-08 18:40:48,150 [main] INFO region_mover - Moving region >> 78c953d53f6498664d9a067701a7e7d7 (42 of 340) to server=3D >> inspur255.deu.edu.cn,60020,1388056934052 >> 2014-01-08 18:40:50,097 [main] INFO region_mover - Moving region >> c621b3bf29262ca5248c03a8d6ebb41e (43 of 340) to server=3D >> inspur253.deu.edu.cn,60020,1388053123213 >> 2014-01-08 18:40:51,652 [main] INFO region_mover - Moving region >> 4dad873a6af4d3a9809339281c3cb34c (44 of 340) to server=3D >> inspur254.deu.edu.cn,60020,1388054917364 >> 2014-01-08 18:40:56,701 [main] INFO region_mover - Moving region >> 0e311941f5ff202bcefe57aa4079a188 (45 of 340) to server=3D >> inspur253.deu.edu.cn,60020,1388053123213 >> 2014-01-08 18:40:58,632 [main] INFO region_mover - Moving region >> 3a955559fb65caf32a05e18b8b6b93f8 (46 of 340) to server=3D >> inspur308.deu.edu.cn,60020,1388059770705 >> 2014-01-08 18:41:02,127 [main] INFO region_mover - Moving region >> 34f3ef516fe6fba940eeb0902b9acd3d (47 of 340) to server=3D >> inspur254.deu.edu.cn,60020,1388054917364 >> 2014-01-08 18:41:03,689 [main] INFO region_mover - Moving region >> ac201a56d80f13ca5357d474578a91c2 (48 of 340) to server=3D >> inspur308.deu.edu.cn,60020,1388059770705 >> 2014-01-08 18:41:05,669 [main] INFO region_mover - Moving region >> 812912be704946d24c5f1b5e3184b2f5 (49 of 340) to server=3D >> inspur253.deu.edu.cn,60020,1388053123213 >> >> I run =E2=80=98du' command to check the last region , which has not a= ny data. >> hadoop@inspur249:~/hbase$ hdfs dfs -du -s -h >> /hbase/test/812912be704946d24c5f1b5e3184b2f5/* >> 486 /hbase/test/812912be704946d24c5f1b5e3184b2f5/.regioninfo >> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/body >> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/meta >> >> hadoop version is cdh4.2.1 and hbase is 0.94 >> >> Thanks! >> Best Regards~ >> Xiyi >> >> > > --047d7b6d8a3ac4220904ef812368--