hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruben Quintero <rfq_...@yahoo.com>
Subject Re: HBase & MapReduce & Zookeeper
Date Thu, 28 Jul 2011 21:28:26 GMT
Yes, that's the connection leak.

Use deleteAllConnections(true), and it will close all open connections.

- Ruben




________________________________
From: Andre Reiter <a.reiter@web.de>
To: user@hbase.apache.org
Sent: Thu, July 28, 2011 4:55:52 PM
Subject: Re: HBase & MapReduce & Zookeeper

i guess, i know the reason, why  HConnectionManager.deleteConnection(conf, 
true); does not work for me

in the MR job im using TableInputFormat, if you have a look at the source code 
in the method
public void setConf(Configuration configuration)
there is a line creating the HTable like this :

...
setHTable(new HTable(new Configuration(conf), tableName));
...

now the HTable is using a new configuration, why???

HConnectionManager holds all  Connections in a map with the config as a key:
private static final Map<Configuration, HConnectionImplementation> 
HBASE_INSTANCE = ...

Configuration does not override the equals and hash methods, so the HConnection 
created by the table using the new Configuration can not be referenced...


just looking at executed code using debugger, i found out, that there is even 
one place more, where another new configuration is created and used:
org.apache.hadoop.mapred.JobClient.submitJobInternal

...
JobContext context = new JobContext(jobCopy, jobId);
...

the JobContext constructor creates a new Configuration object: new 
org.apache.hadoop.mapred.JobConf(conf)

so there is no real chance for me to get the configuration, which was used 
during the call to HConnectionManager.getConnection(conf)



that is why HConnectionManager.deleteConnection(conf, true); does not work for 
me, or am i completely wrong???


just going crazy with that! i'm not doing anything very special, all what i 
want, is to run a MR job out of my application, and NOT by calling it from the 
shell like this:
./bin/hadoop jar /tmp/my.jar package.OurClass
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message