hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From TDLN <diamond...@gmail.com>
Subject Re: Dynamic configuration properties
Date Thu, 29 Jun 2006 18:45:29 GMT
My proposed solution will not help you and I don't know how else to
accomplish this.

Anybody else?

Rgrds, Thomas

On 6/29/06, Enrico Triolo <enrico.triolo@gmail.com> wrote:
> Thank you Thomas, but (if I don't miss something in your suggestion)
> my question is somewhat different. In particular, I can't know in
> advance what values can assume my properties.
>
> Let's say I have a method in my class that starts up nutch and needs
> to pass runtime parameters to my indexing plugin; at the moment, the
> only way I found to accomplish this is something like:
>
> class NutchTest {
>
>    private Configuration conf;
>
>    public NutchTest() {
>       conf = NutchConfiguration.create();
>    }
>
>    public myMethod(int propertyValue) {
>       //set runtime value
>       this.conf.setInt("propertyName", propertyValue);
>       //write an alternate config file with the property just set
>       this.conf.write(new FileOutputStream("myConfig.xml"));
>       //instantiate JobConf that will be passed to nutch
>       JobConf job = new JobConf(this.conf);
>       //add my config file to the resources handled by the JobConf
>       job.addDefaultResource(new Path("myConfig.xml"));
>
>      //start up nutch
>      this.startNutch(job);
>    }
>
>    private void startNutch(JobConf job) {
>       //Let's say this acts as Crawl.main method, with the exception
> that 'job' is used...
>    }
>
> }
>
> This way, in my plugin I can get the property value, and everything
> works as expected.
> I don't really like this solution, especially because the 'write'
> method creates a file with all the properties defined in the
> hadoop-default.xml and hadoop-site.xml along with the ones defined in
> my class.
> It could be a better solution if there could be a way of adding only
> new properties, but the Configuration constructor automatically adds
> hadoop-default and hadoop-site to the default resources. So, I patched
> it and added an alternate constructor that doesn't load default
> resources.
>
> I was just wondering if there is a best-practice for this kind of
> problem or if I missed something in the hadoop API  that can simplify
> this task...
>
> Enrico
>
> On 6/28/06, TDLN <diamond108@gmail.com> wrote:
> > On 0.8-dev the following works for me.
> >
> > 1) add the config property to nutch-site.xml using this format:
> >
> > <property>
> >   <name>propertyName</name>
> >   <value>propertyValue</value>
> >   <description></description>
> > </property>
> >
> > 2) implement org.apache.hadoop.conf.Configurable Interface -> all
> > Filters IndexingFilter, ParseFilter extend this Interface already.
> >
> > private Configuration configuration;
> >
> > public Configuration getConf() {
> >         return configuration;
> > }
> >
> > public void setConf(Configuration configuration) {
> >         this.configuration = configuration;
> > }
> >
> >
> > 3) now get your property:
> >
> > String propertyVal = configuration.get("propertyName");
> >
> > HTH Thomas
> >
> >
> >
> >
> > On 6/28/06, Enrico Triolo <enrico.triolo@gmail.com> wrote:
> > > Hi all.
> > > I embedded nutch in my application and implemented an indexing plugin
> > > to add custom index fields. I need to pass some runtime parameters to
> > > the plugin, so I was wondering if it is possible to set additional
> > > configuration properties at runtime through the Configuration.setXXX
> > > methods.
> > >
> > > In my experiments I couldn't get this to work, so at the moment I'm
> > > just creating a configuration file and adding it with
> > > addDefaultResource each time I run nutch in my application.
> > >
> > > Bye,
> > > Enrico
> > >
> >
>

Mime
View raw message