hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samir Ahmic <ahmic.sa...@gmail.com>
Subject Re: meta server hungs ?
Date Mon, 05 May 2014 08:50:28 GMT
Hi,
This exception:
****
exception=java.net.SocketTimeoutException: Call to
hadoop77/192.168.1.87:60020 failed because java.net.SocketTimeoutException:
60000 millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.1.123:33117
remote=hadoop77/192.168.1.87:60020]
*****
shows that there is connection timeout between master server and
regionserver (hadoop77/192.168.1.87:60020) that is hosting 'meta' table.
Real question is what is causing this timeout?  In my experience it can be
by few things causing this type of timeout. I would suggest that you check
hadoop77/192.168.1.87 <http://192.168.1.87:60020/> Garbage Collection,
memory,  network, CPU disks and i'm sure you will find cause of timeout.
You can us some diagnostic tools like vmstat, sar, iostat to check your
sistem and you can use jstat to check GC and some other JVM stuff.

Regards
Samir




On Mon, May 5, 2014 at 10:14 AM, sunweiwei <sunww@asiainfo-linkage.com>wrote:

> Hi
>
> I'm using hbase0.96.0.
>
> I found client can't put data suddenly  and  hmaster hungs. Then I shutdown
> the hmaster and start a new hmaster, then  the client back to normal.
>
>
>
> I found this logs in the new hmaster . It seem like meta server hungs and
> hmaster stop the meta server.
>
> 2014-04-29 15:32:21,530 INFO  [master:hadoop1:60000]
> catalog.CatalogTracker:
> Failed verification of hbase:meta,,1 at
> address=hadoop77,60020,1396606457005,
> exception=java.net.SocketTimeoutException: Call to
> hadoop77/192.168.1.87:60020 failed because
> java.net.SocketTimeoutException:
> 60000 millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.1.123:33117
> remote=hadoop77/192.168.1.87:60020]
>
> 2014-04-29 15:32:21,532 INFO  [master:hadoop1:60000] master.HMaster:
> Forcing
> expire of hadoop77,60020,1396606457005
>
>
>
> I can't find why meta server hungs .I found this in meta server log
>
> 2014-04-29 13:53:55,637 INFO  [regionserver60020.leaseChecker]
> regionserver.HRegionServer: Scanner 8206938292079629452 lease expired on
> region hbase:meta,,1.1588230740
>
> 2014-04-29 13:53:56,632 INFO  [regionserver60020.leaseChecker]
> regionserver.HRegionServer: Scanner 1111451530521284267 lease expired on
> region hbase:meta,,1.1588230740
>
> 2014-04-29 13:53:56,733 INFO  [regionserver60020.leaseChecker]
> regionserver.HRegionServer: Scanner 516152687416913803 lease expired on
> region hbase:meta,,1.1588230740
>
> 2014-04-29 13:53:56,733 INFO  [regionserver60020.leaseChecker]
> regionserver.HRegionServer: Scanner -2651411216936596082 lease expired on
> region hbase:meta,,1.1588230740
>
>
>
>
>
> any suggestion will be appreciated. Thanks.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message