hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Modify hadoop settings without restart
Date Wed, 13 Jun 2012 14:50:52 GMT
That is a per job property and you can raise it when submitting a job itself.

You can pass it via -D args (See
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html)
or via a conf.set(name, value) from inside the code. Mostly, any prop
that is not of the format mapred(uce).{jobtracker,tasktracker}.*
aren't going to be requiring a restart of services.

On Wed, Jun 13, 2012 at 6:24 PM, Matthias Zengler
<matthias.zengler@googlemail.com> wrote:
> I want to change the mapreduce.(map|reduce).java.opts for a bigger heap
> space of my map and reduce parts.
>
> 2012/6/13 Harsh J <harsh@cloudera.com>
>
>> Matthias,
>>
>> It depends on what config you're trying to change. There are several
>> per-job configs that do not require changes on JT/TT mapred-site.xml
>> and can be passed via -D parameters to a job's configuration directly
>> from CLI (if you use the Tool and ToolRunner methods of writing a
>> driver), or manually via the JobConf/Job.getConfiguration() objects
>> inside the code.
>>
>> So question to you: Which set of configs are you wishing to change
>> every job run? Cause only config changes that apply to JT or TTs need
>> those services to be restarted, the rest can be applied on a per job
>> basis, without requiring a restart of anything.
>>
>> On Wed, Jun 13, 2012 at 5:50 PM, Matthias Zengler
>> <matthias.zengler@googlemail.com> wrote:
>> > Hi,
>> >
>> > I've got a question regarding hadoop configuration. Is it possible to
>> pass
>> > configuration parameters on job start up?
>> > Something like that:
>> >
>> > hadoop -HADOOP_HEAPSIZE=4G jar some.jar some.class.to.execute param1
>> param2
>> >
>> > Or do I have to restart the hadoop cluster every time I want to change
>> > something even if it is just for a specific job or workflow?
>> > We have some jobs running which needs a lot of time and we want to start
>> > another one with a slightly different configuration because it needs more
>> > memory to finish.
>> > We are using CDH3.
>> >
>> > Greetings,
>> > Mat
>>
>>
>>
>> --
>> Harsh J
>>



-- 
Harsh J

Mime
View raw message