hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifeng Jiang <uprushwo...@gmail.com>
Subject Re: bulk load issues with negotiated timeout
Date Mon, 07 May 2012 11:31:43 GMT
Beside the HBase side configuration, you need to add the following to your ZOOKEEPER/conf/zoo.cfg,
and then restart ZooKeeper will allow you to change the 40 secs.
maxSessionTimeout= 120000

Details can be found here.
http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
"The client sends a requested timeout, the server responds with the timeout that it can give
the client. "

The 40 secs timeout is because, ZK's tickTime is set to 2 secs by default, and a maximum timeout
of 20 times the tickTime that ZK server will response.

-Yifeng

On May 7, 2012, at 2:07 PM, Billy Pearson wrote:

> I am running test on the bulk load option on hbase I run the map and everything runs
fine for small data but I tryto load 3-500MB of data and I get timed out and it tries to load
the data several times
> I found the problem just can not find how to fix it how to do you change the negotiated
timeout = 40000 to something else
> 
> I tried to do it in the job config before I load the data to a high number to see if
it helped
> conf.set("zookeeper.session.timeout", "1210000");
> but that does not seams to change the 40 sec timer below.
> 
> So the question is what option changes the 40 secs and where does it go in the conf in
zookeeper/hbase/ or hadoop conf file?
> also can I change it just for this job or does it have to be changed cluster wide?
> 
> 12/05/06 23:54:23 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node2:2181,node1:2181
sessionTimeout=180000 watcher=hconnection
> 12/05/06 23:54:23 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/10.0.0.71:2181
> 12/05/06 23:54:23 INFO zookeeper.ClientCnxn: Socket connection established to node1/10.0.0.71:2181,
initiating session
> 12/05/06 23:54:23 INFO zookeeper.ClientCnxn: Session establishment complete on server
node1/10.0.0.71:2181, sessionid = 0x37100d3f3000f3, negotiated timeout = 40000
> 12/05/06 23:54:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node2:2181,node1:2181
sessionTimeout=1210000 watcher=hconnection
> 12/05/06 23:54:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node2/10.0.0.72:2181
> 12/05/06 23:54:24 INFO zookeeper.ClientCnxn: Socket connection established to node2/10.0.0.72:2181,
initiating session
> 12/05/06 23:54:24 INFO zookeeper.ClientCnxn: Session establishment complete on server
node2/10.0.0.72:2181, sessionid = 0x137100d3f2600ee, negotiated timeout = 40000
> 
> 
> code used to load after the mr job is done
> 
> conf.set("zookeeper.session.timeout", "1210000");
> conf.set("hbase.zookeeper.property.tickTime", "6000");
> HTable table = new HTable("Repo");
> LoadIncrementalHFiles lh = new LoadIncrementalHFiles(conf);
> lh.doBulkLoad(new Path(otherArgs[1]), table);
> 
> 
> 


Mime
View raw message