hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: How to set mapreduce.input.fileinputformat.split.maxsize for a specific job
Date Sun, 17 May 2015 14:37:09 GMT
What do you think is  the type of the property value that you are trying to
write? Is it string? Or numeric? Have you check the documentation of the
Configuration class that I sent earlier?

There are multiple setXXX methods depending on the type of the property
value being set:
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html#setLong(java.lang.String,
long)


For the other case below, why are you setting the job object (first
parameter) as null?
FileInputFormat.setMaxInputSplitSize(null, 102400);
Check out the documentation here:
http://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html#setMaxInputSplitSize(org.apache.hadoop.mapreduce.Job,
long)

Lastly,
 conf.set("mapreduce.input.fileinputformat.split.maxsize", "102400");
VS.
job.getConfiguration().set("mapreduce.input.fileinputformat.split.maxsize",
"102400");
is just a matter of how you are referencing the configuration object.
Either as its own reference of through chained called from the job object.
That is programming style decision and has no bearing on it.

Regards,
Shahab

On Sun, May 17, 2015 at 10:17 AM, Answer Agrawal <yrsna.tset01@gmail.com>
wrote:

> Thanks,
> Is this the correct way to write ?
> conf.set("mapreduce.input.fileinputformat.split.maxsize", "102400");
> or
> job.getConfiguration().set("mapreduce.input.fileinputformat.split.maxsize",
> "102400");
>
> I think another ways as
> FileInputFormat.setMaxInputSplitSize(null, 102400);
>
> Is this all right ? Are these both solve the same purpose or something
> else ?
>
> Thanks,
>
> On Sat, May 16, 2015 at 8:48 PM, Shahab Yunus <shahab.yunus@gmail.com>
> wrote:
>
>> You can either pass them on as command line argument using -D option.
>> Assuming your job is implementing the standard Tool interface:
>>
>> https://hadoop.apache.org/docs/current/api/org/apache/hadoop/util/Tool.html
>>
>> Or you can set them in the code using the various 'set' methods to set
>> key/value values in the configuration object.
>>
>> ...
>> Job job = Job.getInstance(getConf());
>> job.setJarByClass(MyJob.class);
>>
>> job.getConfiguration().set("<property-name>",<value>);
>> ....
>>
>> Docs for Configuration class:
>> https://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html
>>
>> This will work as long as the property is not marked final
>>
>> Regards,
>> Shahab
>>
>>
>> On Sat, May 16, 2015 at 10:49 AM, Answer Agrawal <yrsna.tset01@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> In xmls configuration file of Hadoop-2.x,
>>> "mapreduce.input.fileinputformat.split.minsize" is given which can be set
>>> but how to set "mapreduce.input.fileinputformat.split.maxsize" in xml file.
>>> I need to set it in my mapreduce code.
>>>
>>> Thanks,
>>>
>>
>>
>

Mime
View raw message