hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lucas Nazário dos Santos <nazario.lu...@gmail.com>
Subject Re: Zookeeper apparently going down
Date Thu, 27 May 2010 18:28:57 GMT
> So I see 2 main issues:

>  - Your master's zookeeper session timed out. Why? Hard to tell with
> those logs since it happened before what you pasted. Very slow IO?

I'm not swapping and I doubt that zookeeper session timed out because of
slow IO since my applications aren't even close to stress the hardware. I've
already followed the instruction in
http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9 to avoid this kind of
problem.

> Swapping + GC?
>  - The your region server seemed to have moved elsewhere, or something
> weird like that. DNS blip? Can't tell from the logs.

Maybe a DNS blip. But how can I confirm it? Logs? I didn't move anything and
as soon as I restarted the cluster things got back on track.

The hbase master log just repeated the text bellow for the last 8 hours
before the crash. The zookeeper and region server logs are clear from
errors.

2010-05-27 08:28:15,118 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-05-27 08:28:15,125 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
10.251.158.224:60020, regionname: -ROOT-,,0, startKey: <>} complete
2010-05-27 08:28:26,379 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.251.158.224:60020,
regionname: .META.,,1, startKey: <>}
2010-05-27 08:28:26,787 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 73 row(s) of meta region {server:
10.251.158.224:60020, regionname: .META.,,1, startKey: <>} complete
2010-05-27 08:28:26,788 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2010-05-27 08:28:32,603 INFO org.apache.hadoop.hbase.master.ServerManager: 1
region servers, 0 dead, average load 75.0
2010-05-27 08:29:15,123 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-05-27 08:29:15,138 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
10.251.158.224:60020, regionname: -ROOT-,,0, startKey: <>} complete
2010-05-27 08:29:26,380 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.251.158.224:60020,
regionname: .META.,,1, startKey: <>}
2010-05-27 08:29:26,738 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 73 row(s) of meta region {server:
10.251.158.224:60020, regionname: .META.,,1, startKey: <>} complete
2010-05-27 08:29:26,738 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2010-05-27 08:29:32,605 INFO org.apache.hadoop.hbase.master.ServerManager: 1
region servers, 0 dead, average load 75.0





On Thu, May 27, 2010 at 12:49 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> From what I see, nothing happened to zookeeper.
>
> What happened:
>
> 1) The master wasn't able to scan the -ROOT- region because the
> connection was refused  (same with .META.)
> 2010-05-27 08:40:44,259 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan ROOT region
> java.io.IOException: Call to /10.251.158.224:60020 failed on local
> exception: java.io.IOException: Connection reset by peer
>
> 2) The master's session with zookeeper was timed out
> 2010-05-27 08:40:46,630 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x128c8b303040000 to sun.nio.ch.SelectionKeyImpl@744e022c
> java.io.IOException: Session Expired
>   at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
>   at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
>   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>
> 3) The master was kicked out of the cluster, tries to re-enter
> 2010-05-27 08:40:46,631 INFO org.apache.hadoop.hbase.master.HMaster: Master
> lost its znode, trying to get a new one
>
> 4) The master was able to win the race the be the main master again
> (easy, there's only 1 machine in your cluster)
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Wrote master address
> 10.251.158.224:60000 to ZooKeeper
>
> 5) This master still isn't able to scan -ROOT-
> 2010-05-27 08:41:44,270 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server:
> 10.251.158.224:60020,
> regionname: -ROOT-,,0, startKey: <>}
>
> So I see 2 main issues:
>
>  - Your master's zookeeper session timed out. Why? Hard to tell with
> those logs since it happened before what you pasted. Very slow IO?
> Swapping + GC?
>  - The your region server seemed to have moved elsewhere, or something
> weird like that. DNS blip? Can't tell from the logs.
>
> > Shouldn't Zookeeper recovery nicely? How can I prevent such error from
> > happening in the future?
>
> Nothing happened to zookeeper. And since you have only 1 machine, even
> if the ZK process did die for some reason, how could it even recover?
> Reliability with ZK is 3 machines and more, nothing can be guaranteed
> with only 1 machine.
>
> Now on how to prevent, we need to understand the root cause of the 2
> issues I listed.
>
> Also, not sure if you saw that, but the first minute in your log is
> out of order. Very apparent with the first two lines.
>
> J-D
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message