Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 33A8892B5 for ; Wed, 29 Feb 2012 19:43:29 +0000 (UTC) Received: (qmail 90026 invoked by uid 500); 29 Feb 2012 19:43:27 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 89992 invoked by uid 500); 29 Feb 2012 19:43:27 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 89983 invoked by uid 99); 29 Feb 2012 19:43:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Feb 2012 19:43:27 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.210.169 as permitted sender) Received: from [209.85.210.169] (HELO mail-iy0-f169.google.com) (209.85.210.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Feb 2012 19:43:19 +0000 Received: by iajr24 with SMTP id r24so6399234iaj.14 for ; Wed, 29 Feb 2012 11:42:58 -0800 (PST) Received-SPF: pass (google.com: domain of jdcryans@gmail.com designates 10.42.154.195 as permitted sender) client-ip=10.42.154.195; Authentication-Results: mr.google.com; spf=pass (google.com: domain of jdcryans@gmail.com designates 10.42.154.195 as permitted sender) smtp.mail=jdcryans@gmail.com; dkim=pass header.i=jdcryans@gmail.com Received: from mr.google.com ([10.42.154.195]) by 10.42.154.195 with SMTP id r3mr1508519icw.36.1330544578699 (num_hops = 1); Wed, 29 Feb 2012 11:42:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=TpKYl2Y2vSewZqtiHQqMRX8TvNcwY4+f+qTz73jY4LM=; b=KnlwsCf+Ag3DYzq5oD8aycBeemyYIsEw8kk/wkWhjbw27GL3RV58y2tBZzAe8J8Ccb JzrSL7LsNBeWWFKIgLgYG4r9zaR4lIu238p2eYieWfQMZfIK48Quk8av0iby3NB3AsIm dW1kXwaW3kQ8YtX1MWzPoyVOewlnhYb2KKHzA= MIME-Version: 1.0 Received: by 10.42.154.195 with SMTP id r3mr1240450icw.36.1330544578590; Wed, 29 Feb 2012 11:42:58 -0800 (PST) Sender: jdcryans@gmail.com Received: by 10.231.3.148 with HTTP; Wed, 29 Feb 2012 11:42:58 -0800 (PST) In-Reply-To: References: Date: Wed, 29 Feb 2012 11:42:58 -0800 X-Google-Sender-Auth: tpyrHWnqc9z91Mtk6jXdzAqyp9U Message-ID: Subject: Re: Problem connecting to region server From: Jean-Daniel Cryans To: user@hbase.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org There's a lot going in there and considering that I don't know if your selection if thread dumps/logs is the right one, my suggestions might be wrong. So in that thread dump the Listener thread is blocked on 0x000000041d964510, have you searched which thread holds it? Most of the time (almost 100% in my experience), getting the socket timeout client-side means you need to look at the "IPC Server handler" threads in the dump since this is where the client queries are processed. Regarding your log, it's getting socket timeouts from the Datanode-side. Were you maxing your disks? What was going there? Hope this helps, J-D On Tue, Feb 28, 2012 at 10:04 PM, Yi Liang wrote: > We're running hbase 0.90.3 with hadoop cdh3u2. Today, we ran into a probl= em > connecting to one region server. > > When running hbase hbck, the following error appeared: > Number of Tables: 16 > Number of live region servers: 20 > Number of dead region servers: 0 > .12/02/29 13:06:58 INFO ipc.HbaseRPC: Problem connecting to server: / > 192.168.201.13:60020 > ERROR: RegionServer: test13.xxx.com,60020,1327993969023 Unable to fetch > region information. java.net.SocketTimeoutException: Call to / > 192.168.201.13:60020 failed on socket timeout exception: > java.net.SocketTimeoutException: 60000 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=3D/192.168.201.13:44956re= mote=3D/ > 192.168.201.13:60020] > > and the final status is INCONSISTENT. We have to kill the RS to recover t= he > status. > > From jstack output of that regionserver process, we saw the thread "IPC > Server listener on 60020" had been blocked. We have tried several times > in several minutes, but the state just kept as BLOCKED: > > "IPC Server listener on 60020" daemon prio=3D10 tid=3D0x00007f983c57a800 > nid=3D0x1b12 waiting for monitor entry [0x00007f98388f4000] > =A0 java.lang.Thread.State: BLOCKED (on object monitor) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.registerChannel(H= BaseServer.java:347) > =A0 =A0 =A0 =A0- waiting to lock <0x000000041d964510> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doAccept(HBaseServer.jav= a:496) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:422= ) > > Had it caused the problem connecting to server? But why had it always bee= n > BLOCKED? > > Following is the RS log between the problem appeared and we killed the > process. > > 2012-02-29 12:06:12,117 INFO org.apache.hadoop.hbase.regionserver.Store: > Started compaction of 3 file(s) in cf=3DIndexInfo =A0into hdfs:// > test02.xxx.com:30070/offline-hbase/News/4dae1f8cd991f17414ca4d86ff0884ad/= .tmp, > seqid=3D423340578, totalSize=3D8.5m > 2012-02-29 12:06:12,118 DEBUG org.apache.hadoop.hbase.regionserver.Store: > Compacting hdfs:// > test02.xxx.com:30070/offline-hbase/News/4dae1f8cd991f17414ca4d86ff0884ad/= IndexInfo/8324806988914852495, > keycount=3D122337, bloomtype=3DNONE, size=3D8.4m > 2012-02-29 12:06:12,118 DEBUG org.apache.hadoop.hbase.regionserver.Store: > Compacting hdfs:// > test02.xxx.com:30070/offline-hbase/News/4dae1f8cd991f17414ca4d86ff0884ad/= IndexInfo/1116030618027381242, > keycount=3D258, bloomtype=3DNONE, size=3D17.7k > 2012-02-29 12:06:12,118 DEBUG org.apache.hadoop.hbase.regionserver.Store: > Compacting hdfs:// > test02.xxx.com:30070/offline-hbase/News/4dae1f8cd991f17414ca4d86ff0884ad/= IndexInfo/3755533953967637627, > keycount=3D372, bloomtype=3DNONE, size=3D25.8k > 2012-02-29 12:06:12,906 INFO org.apache.hadoop.hbase.regionserver.Store: > Completed major compaction of 3 file(s), new file=3Dhdfs:// > test02.xxx.com:30070/offline-hbase/News/4dae1f8cd991f17414ca4d86ff0884ad/= IndexInfo/3731399222200436246, > size=3D8.5m; total size for store is 8.5m > 2012-02-29 12:06:12,906 INFO org.apache.hadoop.hbase.regionserver.HRegion= : > completed compaction on region > News,57addda034c334e4,1313319088489.4dae1f8cd991f17414ca4d86ff0884ad. aft= er > 7sec > 2012-02-29 12:07:26,577 INFO org.apache.hadoop.hdfs.DFSClient: Could not > obtain block blk_1313036207534951503_65938873 from any node: > java.io.IOException: No live nodes contain current block. Will get new > block locations from namenode and retry... > 2012-02-29 12:07:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.49 G= B, > free=3D646.64 MB, max=3D3.12 GB, blocks=3D29616, accesses=3D80631725, > hits=3D60715195, hitRatio=3D75.29%%, cachingAccesses=3D72673671, > cachingHits=3D59497193, cachingHitsRatio=3D81.86%%, evictions=3D3584, > evicted=3D13146860, evictedPerRun=3D3668.208740234375 > 2012-02-29 12:12:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.62 G= B, > free=3D518.67 MB, max=3D3.12 GB, blocks=3D30943, accesses=3D80640574, > hits=3D60722719, hitRatio=3D75.30%%, cachingAccesses=3D72682520, > cachingHits=3D59504717, cachingHitsRatio=3D81.86%%, evictions=3D3584, > evicted=3D13146860, evictedPerRun=3D3668.208740234375 > 2012-02-29 12:15:06,937 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction > started; Attempting to free 319.74 MB of total=3D2.65 GB > 2012-02-29 12:15:06,955 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction > completed; freed=3D319.87 MB, total=3D2.34 GB, single=3D744.45 MB, multi= =3D1.9 GB, > memory=3D0 KB > 2012-02-29 12:17:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.45 G= B, > free=3D692.03 MB, max=3D3.12 GB, blocks=3D28911, accesses=3D80652333, > hits=3D60732703, hitRatio=3D75.30%%, cachingAccesses=3D72694279, > cachingHits=3D59514701, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:18:52,867 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /192.168.201.23:50010 for file > /offline-hbase/News/bec970594146b62ddf8bd450fc654acf/Content/621772312284= 239615 > for block -1994034029269165490:java.net.SocketTimeoutException: 60000 > millis timeout while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=3D/192.168.201.13:48546re= mote=3D/ > 192.168.201.23:50010] > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:1= 64) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.fill(BufferedInputStream.ja= va:218) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:237) > =A0 =A0 =A0 =A0at java.io.DataInputStream.readShort(DataInputStream.java:= 295) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.jav= a:1462) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSCl= ient.java:2024) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2099) > =A0 =A0 =A0 =A0at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounded= RangeFileInputStream.java:101) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read1(BufferedInputStream.j= ava:256) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:317) > =A0 =A0 =A0 =A0at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.jav= a:1442) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1= 299) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Store= FileScanner.java:136) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScann= er.java:96) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.jav= a:77) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1345) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion= .java:2274) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(H= Region.java:1131) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1123= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1107= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2996) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2898) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java= :1630) > =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Sou= rce) > =A0 =A0 =A0 =A0at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039= ) > 2012-02-29 12:18:55,395 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /192.168.201.23:50010 for file > /offline-hbase/News/a959f6488cb5f8a13c5e63e0a149b18b/IndexInfo/5814475919= 152417643 > for block -4812950919171511907:java.net.SocketTimeoutException: 60000 > millis timeout while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=3D/192.168.201.13:48597re= mote=3D/ > 192.168.201.23:50010] > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:1= 64) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.fill(BufferedInputStream.ja= va:218) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:237) > =A0 =A0 =A0 =A0at java.io.DataInputStream.readShort(DataInputStream.java:= 295) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.jav= a:1462) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSCl= ient.java:2024) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2099) > =A0 =A0 =A0 =A0at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounded= RangeFileInputStream.java:101) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read1(BufferedInputStream.j= ava:256) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:317) > =A0 =A0 =A0 =A0at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.jav= a:1442) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1= 299) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Store= FileScanner.java:136) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScann= er.java:96) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.jav= a:77) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1345) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion= .java:2274) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(H= Region.java:1131) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1123= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1107= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2996) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2898) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java= :1630) > =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Sou= rce) > =A0 =A0 =A0 =A0at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039= ) > 2012-02-29 12:18:55,395 INFO org.apache.hadoop.hdfs.DFSClient: Could not > obtain block blk_-4812950919171511907_69296565 from any node: > java.io.IOException: No live nodes contain current block. Will get new > block locations from namenode and retry... > 2012-02-29 12:18:56,459 WARN org.apache.hadoop.hdfs.DFSClient: > DFSOutputStream ResponseProcessor exception =A0for block > blk_-8859664738058583740_69352641java.io.IOException: Bad response 1 for > block blk_-8859664738058583740_69352641 from datanode 192.168.201.23:5001= 0 > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DF= SClient.java:2651) > > 2012-02-29 12:18:56,460 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-8859664738058583740_69352641 bad datanode[1] > 192.168.201.23:50010 > 2012-02-29 12:18:56,460 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-8859664738058583740_69352641 in pipeline > 192.168.201.13:50010, 192.168.201.23:50010, 192.168.201.15:50010: bad > datanode 192.168.201.23:50010 > 2012-02-29 12:18:56,460 INFO org.apache.hadoop.ipc.Client: Retrying conne= ct > to server: /192.168.201.13:50020. Already tried 0 time(s). > 2012-02-29 12:19:01,111 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /192.168.201.23:50010 for file > /offline-hbase/News/1ebe9ad2bbad7c8e584bce4cc22f8278/BasicInfo/7333739767= 200146616 > for block 7663668744337108616:java.net.SocketTimeoutException: 60000 mill= is > timeout while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=3D/192.168.201.13:48682re= mote=3D/ > 192.168.201.23:50010] > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:1= 64) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.fill(BufferedInputStream.ja= va:218) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:237) > =A0 =A0 =A0 =A0at java.io.DataInputStream.readShort(DataInputStream.java:= 295) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.jav= a:1462) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSCl= ient.java:2024) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2099) > =A0 =A0 =A0 =A0at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bounded= RangeFileInputStream.java:101) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read1(BufferedInputStream.j= ava:256) > =A0 =A0 =A0 =A0at java.io.BufferedInputStream.read(BufferedInputStream.ja= va:317) > =A0 =A0 =A0 =A0at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.jav= a:1442) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1= 299) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(Store= FileScanner.java:136) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScann= er.java:96) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.jav= a:77) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1345) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion= .java:2274) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(H= Region.java:1131) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1123= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1107= ) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2996) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2898) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java= :1630) > =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Sou= rce) > =A0 =A0 =A0 =A0at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25) > =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039= ) > 2012-02-29 12:19:02,772 WARN org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > Responder, call get([B@665bd4b4, row=3D64d3ef8647252a85, maxVersions=3D1, > cacheBlocks=3Dtrue, timeRange=3D[0,9223372036854775807), > families=3D{(family=3DBasicInfo, columns=3D{EntryPage, HostName, SID, URL= }), > (family=3DContent, columns=3D{ArchItem, ContentGroup, HTTPBody, TagInfo}}= ) from > 192.168.201.27:32866: output error > 2012-02-29 12:19:02,772 WARN org.apache.hadoop.ipc.HBaseServer: IPC Serve= r > handler 58 on 60020 caught: java.nio.channels.ClosedChannelException > =A0 =A0 =A0 =A0at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133) > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.ja= va:324) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1389) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:134= 1) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseSe= rver.java:727) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.j= ava:792) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083= ) > > 2012-02-29 12:22:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:27:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:32:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:37:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:42:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:47:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:52:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 12:57:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 13:05:35,372 DEBUG > org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000m= s > elapsed > 2012-02-29 13:05:35,377 INFO > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using > syncFs -- HDFS-200 > 2012-02-29 13:07:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > 2012-02-29 13:12:50,103 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=3D2.46 G= B, > free=3D678.89 MB, max=3D3.12 GB, blocks=3D29075, accesses=3D80653112, > hits=3D60733340, hitRatio=3D75.30%%, cachingAccesses=3D72695058, > cachingHits=3D59515338, cachingHitsRatio=3D81.86%%, evictions=3D3585, > evicted=3D13150645, evictedPerRun=3D3668.2412109375 > Wed Feb 29 13:14:07 CST 2012 Killing regionserver > > Thanks, > Yi