hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Bug when reusing org.apache.hadoop.conf.Configuration in HbaseConfiguration
Date Wed, 13 Oct 2010 22:06:17 GMT
On Wed, Oct 13, 2010 at 10:09 AM, Henning Blohm
<henning.blohm@zfabrik.de> wrote:
> Hi,
>  I used
> Configuration cfg = new Configuration();
> cfg.addResource(InputStream)
> on org.apache.hadoop.conf.Configuration to create hadoop configuration.
> That configuration is
> used to construct HbaseConfiguration objects several times later on:
> HbaseConfiguration hbcfg = new HbaseConfiguration(cfg)

In TRUNK HBaseConfiguration is deprecated.  Do you see same phenomena
when you do HBaseConfiguration.create(cfg)?

See below...

> At the second this leads to an error as the passed on configuraiton
> object
> is asked to load its properties again, trying to read from the
> inputstream once more
> (as the stream is memorized!!).
> The fact that the properties of the passed on configuration object have
> been reset somewhat
> surprising: When using the hbase configuration object constructed above
> in HbaseAdmin, this
> will eventually load JobConf which has a static initializer that calls
> Configuration.addDefaultResource("mapred-default.xml");
> which in turn goes through the Configuration.REGISTRY which holds on to
> any previously
> not yet collected Configuration objects to (hold your breath!) force
> them to reload their configuration
> by calling reloadConfiguration that (now we are there) sets
> properties=null.
> Did anybody follow that....
> It seems there is somewhat surprising side effects in hadoop/hbase
> configuration handling.
> Wouldn't it be better to have the default resource (pragmatically)
> defined once in Configuration
> and not (even think about) touch already instantiated config objects
> later on?

Yes.  That sounds completely reasonable.

See if HBaseConfiguration.create(cfg) gives you the same issue.  If
so, then its the way Hadoop Configuration works currently.   I haven't
spent time on it in a while but I remember getting into interesting
scenarios loading properties.  If you can confine the issue some, lets
file an issue up in hadoop common?

Thanks Henning,

View raw message