hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Su <scarcer...@gmail.com>
Subject Hbase connection closed when query multiple complicated hql with hive+hbase integration
Date Wed, 07 Nov 2012 03:07:37 GMT
Hi, all.

I have a hive + hbase integration cluster. I met a performance issue.
I executed some complicated hql, but only the first one is actually running.
The rests are showed "running" on the job track web ui, but task
attempt failed with exceptions below:

java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5117a20
closed
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
	at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
	... 9 more

So actually the rests are queuing.

I guess it might be because the count(?) of hbase handler is too
small,  which handle access request from hive.
So that when a big job occupied all handles the rests have to wait
until the handlers are released.

Is my assumption right? and what settings should I tuning?

Thanks.

-- 

Regards,
Cheng Su

Mime
View raw message