hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sreekanth Ramakrishnan <sreer...@yahoo-inc.com>
Subject Re: Change Map-Red Parameters
Date Fri, 18 Mar 2011 04:44:59 GMT
Hi Adarsh,

Configuration class supports addition of external resources other than default Hadoop MapRed
resources. You can define your own resource file. For instance like mapred-site.xml you can
have myjobs-site.xml. You can define your job specific parameters in the same.

In general mapred-site.xml contains common set of configurations which a job client uses to
submit the given job. Changing it dynamically can potentially cause debugging pains later.
So an ideal way would be defining new configuration file, adding the same to classpath and
programmatically setting the same as resource while creating your job configuration. You can
periodically make changes to the newly defined resource. So if you want to override some mapred
specific parameters in the same, just see to that you load the newly created resources after
mapred resource files are loaded.


On 3/18/11 10:03 AM, "Adarsh Sharma" <adarsh.sharma@orkash.com> wrote:

Dear all,

Is it possible in Hadoop Map-Reduce Framework to change the job
execution parameters during runtime.

I know we can set through job.setNumTasks and many parameters but this
remains only for that job. In Hadoop Pipes we can set it through -D
parameter. But the problem is other.

What I want is to run a MAp-Reduce program that changes the
mapred-site.xml parameters and the changes got reflected in the next jobs.

Please guide me if there is a way to do this.

Thanks & best Regards
Adarsh Sharma

Sreekanth Ramakrishnan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message