hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop doesnt use Replication Level of Namenode
Date Tue, 13 Sep 2011 04:02:45 GMT

There is no current way to 'fetch' a config at the moment. You have
the NameNode's config available at NNHOST:WEBPORT/conf page which you
can perhaps save as a resource (dynamically) and load into your
Configuration instance, but apart from this hack the only other ways
are the ones Bharath mentioned. This might lead to slow start ups of
your clients, but would give you the result you want.

You can also write a simple service that 'serves' you the latest,
loaded configs from its location (you can use ZK for availability of
such a service). And all your code can use this location to get its
configuration objects from. This is another centralized way, I
suppose. IIRC someone on the list had something like this in place as

Take it as something like flexibility. Client-side configurations are
flexible so that each client node can submit with their own
properties, based on its needs. You get finer control, but I agree
that at least a base-minimum server config could get auto-discovered.
That feature isn't present yet. Feel free to open or comment on
existing JIRAs surrounding this, and if possible, patches welcome! :)

On Tue, Sep 13, 2011 at 3:31 AM, Ralf Heyde <ralf.heyde@gmx.de> wrote:
> Hello,
> I've writed an HDFS Client which works pretty well.
> But . on my Namenode I configured a replication leven of 2 . on my Client -
> the config - hold a value of 1.
> If I now write a file from my HDFSClient to the HDFS it gets the
> replication-value of 1.
> I know that I can manually put the property to 2 in my Java Code - but:
> Is there any possibility OR workaround to use/get the configuration of the
> Namenode?
> My current workaround is the copying of all configuration files from the
> namenode to the local client - but that's *** .
> Does anybody has an idea?
> Thanks, Ralf

Harsh J

View raw message