hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: JVM option
Date Fri, 18 Apr 2014 21:48:04 GMT
HADOOP_HEAPSIZE and HADOOP_OPTS are not hadoop properties. These seem to
be environment variables of the shell script.

Try this with a space between -D and the property name (
-Dmapred.child.java.opts)

See the properties available (for mrv1) here:
http://hadoop.apache.org/docs/r1.0.4/mapred-default.html

or this:
http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

As for passing system properties, the option that I know of (others can
chime in if they know something better) is to pass the hadoop property and
then in the java code, set the system property form the configuration
object.

Regards,
Shahab


On Fri, Apr 18, 2014 at 3:20 PM, Andy Srine <andy.srine@gmail.com> wrote:

> I guess then the question is, how do I set the (system?) property and what
> property exactly is this?
>
> "export HADOOP_HEAPSIZE=4096" - Works from the environment.
>
> And none of the following seem to work from the command line.
> -Xmx4G
>
> -Dmapred.child.java.opts=-Xmx4G
>
> -DHADOOP_HEAPSIZE=4096
>
> -DHADOOP_OPTS=-Xmx4G
>
>
> I tried this from Java code as well and no luck.
>
> System.setProperty("HADOOP_HEAPSIZE", "4096");
>
>
> What am I missing?
>
>
> Thanks,
>
> Andy
>
>
>
>
> On Fri, Apr 18, 2014 at 11:28 AM, Shahab Yunus <shahab.yunus@gmail.com>wrote:
>
>> You can pass hadoop conf properties through the -D option. Have you seen
>> this?
>>
>> http://stackoverflow.com/questions/15490090/how-to-specify-system-property-in-hadoop-except-modify-hadoop-env-sh
>>
>> This is not for system properties. The assumption is that you want to
>> specify hadoop conf property through the command line.
>>
>> Regards,
>> Shahab
>>
>>
>> On Fri, Apr 18, 2014 at 2:12 PM, Andy Srine <andy.srine@gmail.com> wrote:
>>
>>> Thanks for the tip. May be it's a different setting I need. Let me
>>> explain the problem.
>>>
>>> I get an heap error on the command line - "Exception in thread "main"
>>> java.lang.OutOfMemoryError: Java heap space". It woks when I set "export
>>> HADOOP_HEAPSIZE=4096" in my env. But how can I set it via the hadoop
>>> command line (hadoop jar ...)? Next choice would be to do it
>>> programmatically in Java using hadoop conf. I can't really change the XML
>>> files across the cluster.
>>>
>>> Regards,
>>> Andy
>>>
>>>
>>> On Thu, Apr 17, 2014 at 6:39 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>
>>>> Do you want add "-Xmx4g" to your MR tasks?  if so, just add it as
>>>> "mapred.child.java.opts" in the mapred-site.xml
>>>>
>>>>
>>>> On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine <andy.srine@gmail.com>wrote:
>>>>
>>>>> Quick question. How would I pass the following JVM option to the
>>>>> Hadoop command line?
>>>>>
>>>>> "-Xmx4G"
>>>>>
>>>>> hadoop jar <jar file> <class_name> <args>
>>>>>
>>>>> Thanks,
>>>>> Andy
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andy
>>>
>>>
>>
>
>
> --
> Thanks,
> Andy
>
>

Mime
View raw message