hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: massive zk expirations under heavy network load
Date Wed, 20 Apr 2011 19:16:02 GMT
Kazuki-san,

Setting the ZK timeout to a large value will stop the expirations but may not provide sufficiently
fast failure detection for your use case of course.

However if even Ganglia stops working during a large mapreduce job, I think you need to question
the adequacy of the network hardware.

   - Andy

> From: Kazuki Ohta <kazuki.ohta@gmail.com>
> Subject: massive zk expirations under heavy network load
> To: user@hbase.apache.org
> Cc: kazuki.ohta@gmail.com
> Date: Wednesday, April 20, 2011, 11:41 AM
> Hi,
> 
> I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
> The configuraiton is below.
> 
> hdp0: zk + master + region + nn + dn + jt + tt
> hdp1: zk + master + region + snn + dn + tt
> hdp2: zk + region + dn + tt
> hdp3 to hdp15: region + dn + tt
> 
> Usually, it works really well. But once the user throws MapReduce
> job which requires massive network transfer in the shuffle phase,
> the master got the zk session timeout exception, and fails-over to
> another master.
> 
> The problem is that shuffle network transfer dominates the switch,
> and important zk packets are not transferred properly at
> that time.
> 
> Even ganglia monitoring seems to stop at that time. And mr task
> attempts also got zk session timeouts and dies altogether (about
> 100 tasks dies at the same time. input and output are both hbase).
> 
> This is the potential problem running MapReduce job alongside
> with HBase. Does anyone know any good solution for this
> phenomenon?



Mime
View raw message