hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Pino <juancitomiguel...@gmail.com>
Subject Re: load properties from property file into Configuration object
Date Sat, 03 Aug 2013 12:46:40 GMT
Thanks Harsh for your reply.
Yes I think I am using the Tool framework ("public class HadoopJob extends
Configured implements Tool" + main function is "System.exit(ToolRunner.run(new
HadoopJob(), args));")

"then you can pass -Dkey=val props directly into a config object at runtime"
I'm a bit confused by this: I can pass -D key1=val1 -D key2=val2 etc. on
the command line, and with the Tool framework, those properties get
automatically loaded in the Configuration object, but if I have many of
these properties, how can I write them into a config file so that these
properties are loaded into the Configuration object ?

Thanks very much,

Juan


On Sat, Aug 3, 2013 at 1:20 PM, Harsh J <harsh@cloudera.com> wrote:

> This is a bit unnecessary to manually do. If you use the Tool
> framework, then you can pass -Dkey=val props directly into a config
> object at runtime. See
> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/Tool.html
> and
> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/GenericOptionsParser.html
>
> On Sat, Aug 3, 2013 at 5:30 PM, Juan Pino <juancitomiguelito@gmail.com>
> wrote:
> > Hi,
> >
> > I am trying to simplify code that loads properties from a property file
> into
> > a Configuration object:
> >
> > Short excerpt of the code:
> >
> > String configFile = args[0];
> > Properties p = new Properties();
> > p.load(new FileInputStream(configFile));
> > Configuration conf = getConf();
> > for (String prop: p.stringPropertyNames()) {
> >   conf.set(prop, p.getProperty(prop));
> > }
> >
> > The context is that my program can be run with an optional argument
> which is
> > a config file as "hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D
> > property2=PROPERTY2 [MYCONFIGFILE]"
> > The format of MYCONFIGFILE is
> > "property3=PROPERTY3<NEWLINE>property4=PROPERTY4 etc."
> >
> > Looking at the Configuration api, I didn't see any method that directly
> > loads properties from a property file.
> > Looking at the Configuration code, Configuration looks like a wrapper
> around
> > Properties, so I thought there was a similar method to Properties.load
> or a
> > method that sets several properties at the same time.
> > Is there a way to do this ?
> >
> > Thanks very much,
> >
> > Juan
> >
> > --
> > In light of the recent NSA scandal, please consider encrypting your
> reply by
> > using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txtIf
> > you use webmail, you may consider mailvelope (http://mailvelope.com/)
> which
> > is a very easy to use plugin available for chrome and probably soon for
> > firefox.
>
>
>
> --
> Harsh J
>



-- 
In light of the recent NSA scandal, please consider encrypting your reply
by using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt If
you use webmail, you may consider mailvelope (http://mailvelope.com/) which
is a very easy to use plugin available for chrome and probably soon for
firefox.

Mime
View raw message