hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enrico Triolo" <enrico.tri...@gmail.com>
Subject Re: Dynamic configuration properties
Date Thu, 29 Jun 2006 08:25:57 GMT
Thank you Thomas, but (if I don't miss something in your suggestion)
my question is somewhat different. In particular, I can't know in
advance what values can assume my properties.

Let's say I have a method in my class that starts up nutch and needs
to pass runtime parameters to my indexing plugin; at the moment, the
only way I found to accomplish this is something like:

class NutchTest {

   private Configuration conf;

   public NutchTest() {
      conf = NutchConfiguration.create();
   }

   public myMethod(int propertyValue) {
      //set runtime value
      this.conf.setInt("propertyName", propertyValue);
      //write an alternate config file with the property just set
      this.conf.write(new FileOutputStream("myConfig.xml"));
      //instantiate JobConf that will be passed to nutch
      JobConf job = new JobConf(this.conf);
      //add my config file to the resources handled by the JobConf
      job.addDefaultResource(new Path("myConfig.xml"));

     //start up nutch
     this.startNutch(job);
   }

   private void startNutch(JobConf job) {
      //Let's say this acts as Crawl.main method, with the exception
that 'job' is used...
   }

}

This way, in my plugin I can get the property value, and everything
works as expected.
I don't really like this solution, especially because the 'write'
method creates a file with all the properties defined in the
hadoop-default.xml and hadoop-site.xml along with the ones defined in
my class.
It could be a better solution if there could be a way of adding only
new properties, but the Configuration constructor automatically adds
hadoop-default and hadoop-site to the default resources. So, I patched
it and added an alternate constructor that doesn't load default
resources.

I was just wondering if there is a best-practice for this kind of
problem or if I missed something in the hadoop API  that can simplify
this task...

Enrico

On 6/28/06, TDLN <diamond108@gmail.com> wrote:
> On 0.8-dev the following works for me.
>
> 1) add the config property to nutch-site.xml using this format:
>
> <property>
>   <name>propertyName</name>
>   <value>propertyValue</value>
>   <description></description>
> </property>
>
> 2) implement org.apache.hadoop.conf.Configurable Interface -> all
> Filters IndexingFilter, ParseFilter extend this Interface already.
>
> private Configuration configuration;
>
> public Configuration getConf() {
>         return configuration;
> }
>
> public void setConf(Configuration configuration) {
>         this.configuration = configuration;
> }
>
>
> 3) now get your property:
>
> String propertyVal = configuration.get("propertyName");
>
> HTH Thomas
>
>
>
>
> On 6/28/06, Enrico Triolo <enrico.triolo@gmail.com> wrote:
> > Hi all.
> > I embedded nutch in my application and implemented an indexing plugin
> > to add custom index fields. I need to pass some runtime parameters to
> > the plugin, so I was wondering if it is possible to set additional
> > configuration properties at runtime through the Configuration.setXXX
> > methods.
> >
> > In my experiments I couldn't get this to work, so at the moment I'm
> > just creating a configuration file and adding it with
> > addDefaultResource each time I run nutch in my application.
> >
> > Bye,
> > Enrico
> >
>

Mime
View raw message