Hello,
I've set up a test HBase+Hadoop cluster yesterday and got the following
error in logs during running MR job (which internally creates HTable for
Reducer):
KeeperErrorCode = ConnectionLoss for /hbase
Then I went to Zookeeper logs and found this:
2011-03-24 22:41:49,884 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/x.x.x.x - max is 10
As far as I know HBase configured to initiate up to 30 connections by
default, and maxClientCnxns for Zookeeper was meant to be 30 as well. Is
this a bug that I have it set to 10 by default? I use a version:
zookeeper-3.3.2-CDH3B4. Sorry if this issue is specific to CDH (then I
probably should post it in the correspondent ML).
After setting maxClientCnxns explicitly to 30 in zoo.cfg things work well.
Just to prevent some Qs, I have this in code:
when starting job:
Configuration conf = HBaseConfiguration.create();
Job job = new Job(conf, aggRawDataJobClass.getName());
TableMapReduceUtil.initTableMapperJob(rawLogsTableName, scan,
mapperClass, ImmutableBytesWritable.class, mappedAggregateClass,
job);
in reducer:
public static class CustomReducer<T> extends
TableReducer<ImmutableBytesWritable, T, Writable> {
@Override
protected void setup(Context context) throws IOException,
InterruptedException {
...
hTable = new HTable(context.getConfiguration(), tableName);
...
}
}
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/
|