hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Pino <juancitomiguel...@gmail.com>
Subject Re: load properties from property file into Configuration object
Date Sat, 03 Aug 2013 13:51:59 GMT
I guess -conf won't support a config file in property format rather than
xml format ?


On Sat, Aug 3, 2013 at 2:23 PM, Harsh J <harsh@cloudera.com> wrote:

> Yes, you could use an XML file for that purpose (i.e. lots of config
> options) and pass it along with -conf.
>
> On Sat, Aug 3, 2013 at 6:32 PM, Juan Pino <juancitomiguelito@gmail.com>
> wrote:
> > Ok sorry, I think I got it: I would need to run my program like this:
> >
> > hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D property2=PROPERTY2
> -conf
> > MYCONFIGFILE"
> >
> > Juan
> >
> >
> > On Sat, Aug 3, 2013 at 1:46 PM, Juan Pino <juancitomiguelito@gmail.com>
> > wrote:
> >>
> >> Thanks Harsh for your reply.
> >> Yes I think I am using the Tool framework ("public class HadoopJob
> extends
> >> Configured implements Tool" + main function is
> >> "System.exit(ToolRunner.run(new HadoopJob(), args));")
> >>
> >> "then you can pass -Dkey=val props directly into a config object at
> >> runtime"
> >> I'm a bit confused by this: I can pass -D key1=val1 -D key2=val2 etc. on
> >> the command line, and with the Tool framework, those properties get
> >> automatically loaded in the Configuration object, but if I have many of
> >> these properties, how can I write them into a config file so that these
> >> properties are loaded into the Configuration object ?
> >>
> >> Thanks very much,
> >>
> >> Juan
> >>
> >>
> >> On Sat, Aug 3, 2013 at 1:20 PM, Harsh J <harsh@cloudera.com> wrote:
> >>>
> >>> This is a bit unnecessary to manually do. If you use the Tool
> >>> framework, then you can pass -Dkey=val props directly into a config
> >>> object at runtime. See
> >>>
> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/Tool.html
> >>> and
> >>>
> http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/util/GenericOptionsParser.html
> >>>
> >>> On Sat, Aug 3, 2013 at 5:30 PM, Juan Pino <juancitomiguelito@gmail.com
> >
> >>> wrote:
> >>> > Hi,
> >>> >
> >>> > I am trying to simplify code that loads properties from a property
> file
> >>> > into
> >>> > a Configuration object:
> >>> >
> >>> > Short excerpt of the code:
> >>> >
> >>> > String configFile = args[0];
> >>> > Properties p = new Properties();
> >>> > p.load(new FileInputStream(configFile));
> >>> > Configuration conf = getConf();
> >>> > for (String prop: p.stringPropertyNames()) {
> >>> >   conf.set(prop, p.getProperty(prop));
> >>> > }
> >>> >
> >>> > The context is that my program can be run with an optional argument
> >>> > which is
> >>> > a config file as "hadoop jar MYJAR MYCLASS -D property1=PROPERTY1 -D
> >>> > property2=PROPERTY2 [MYCONFIGFILE]"
> >>> > The format of MYCONFIGFILE is
> >>> > "property3=PROPERTY3<NEWLINE>property4=PROPERTY4 etc."
> >>> >
> >>> > Looking at the Configuration api, I didn't see any method that
> directly
> >>> > loads properties from a property file.
> >>> > Looking at the Configuration code, Configuration looks like a wrapper
> >>> > around
> >>> > Properties, so I thought there was a similar method to
> Properties.load
> >>> > or a
> >>> > method that sets several properties at the same time.
> >>> > Is there a way to do this ?
> >>> >
> >>> > Thanks very much,
> >>> >
> >>> > Juan
> >>> >
> >>> > --
> >>> > In light of the recent NSA scandal, please consider encrypting your
> >>> > reply by
> >>> > using my public key available at
> http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
> >>> > If
> >>> > you use webmail, you may consider mailvelope (http://mailvelope.com/
> )
> >>> > which
> >>> > is a very easy to use plugin available for chrome and probably soon
> for
> >>> > firefox.
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >>
> >>
> >>
> >> --
> >> In light of the recent NSA scandal, please consider encrypting your
> reply
> >> by using my public key available at
> http://mi.eng.cam.ac.uk/~jmp84/pgp.txt
> >> If you use webmail, you may consider mailvelope (http://mailvelope.com/
> )
> >> which is a very easy to use plugin available for chrome and probably
> soon
> >> for firefox.
> >
> >
> >
> >
> > --
> > In light of the recent NSA scandal, please consider encrypting your
> reply by
> > using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txtIf
> > you use webmail, you may consider mailvelope (http://mailvelope.com/)
> which
> > is a very easy to use plugin available for chrome and probably soon for
> > firefox.
>
>
>
> --
> Harsh J
>



-- 
In light of the recent NSA scandal, please consider encrypting your reply
by using my public key available at http://mi.eng.cam.ac.uk/~jmp84/pgp.txt If
you use webmail, you may consider mailvelope (http://mailvelope.com/) which
is a very easy to use plugin available for chrome and probably soon for
firefox.

Mime
View raw message