hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: setting mapred.map.child.java.opts not working
Date Thu, 12 Jan 2012 06:49:18 GMT
@Koji - Ah, I had not checked the sources, rather just the JIRA, which I've updated now since
there is indeed an svn commit listed there. Thanks for correcting me there!

@George - I do not think it matters if these are present in default, but they should be present,
yes. Mind filing a JIRA if this is still not the case in 0.23+?

@Vinod - Do you see these props you set appear in your submitted job.xml? What does your job
do exactly, and how much of RAM:Slots are configured/available for your slave machine?

On 12-Jan-2012, at 11:04 AM, George Datskos wrote:

> Koji, Harsh
> 
> mapred-478 seems to be in v1, but those new settings have not yet been added to mapred-default.xml.
 (for backwards compatibility?)
> 
> 
> George
> 
> On 2012/01/12 13:50, Koji Noguchi wrote:
>> Hi Harsh,
>> 
>> Wasn't MAPREDUCE-478 in 1.0 ?  Maybe the Jira is not up to date.
>> 
>> Koji
>> 
>> 
>> On 1/11/12 8:44 PM, "Harsh J"<harsh@cloudera.com>  wrote:
>> 
>>> These properties are not available on Apache Hadoop 1.0 (Formerly
>>> known as 0.20.x). This was a feature introduced in 0.21
>>> (https://issues.apache.org/jira/browse/MAPREDUCE-478), and is
>>> available today on 0.22 and 0.23 line of releases.
>>> 
>>> For 1.0/0.20, use "mapred.child.java.opts", that applies to both map
>>> and reduce commonly.
>>> 
>>> Would also be helpful if you can tell us what doc guided you to use
>>> these property names instead of the proper one, so we can fix it.
>>> 
>>> On Thu, Jan 12, 2012 at 8:44 AM, T Vinod Gupta<tvinod@readypulse.com> 
wrote:
>>>> Hi,
>>>> Can someone help me asap? when i run my mapred job, it fails with this
>>>> error -
>>>> 12/01/12 02:58:36 INFO mapred.JobClient: Task Id :
>>>> attempt_201112151554_0050_m_000071_0, Status : FAILED
>>>> Error: Java heap space
>>>> attempt_201112151554_0050_m_000071_0: log4j:ERROR Failed to flush writer,
>>>> attempt_201112151554_0050_m_000071_0: java.io.IOException: Stream closed
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:44)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:139)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:58)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:260)
>>>> attempt_201112151554_0050_m_000071_0:   at
>>>> org.apache.hadoop.mapred.Child$2.run(Child.java:142)
>>>> 
>>>> 
>>>> so i updated my mapred-site.xml with these settings -
>>>> 
>>>>  <property>
>>>>    <name>mapred.map.child.java.opts</name>
>>>>    <value>-Xmx2048M</value>
>>>>  </property>
>>>> 
>>>>  <property>
>>>>    <name>mapred.reduce.child.java.opts</name>
>>>>    <value>-Xmx2048M</value>
>>>>  </property>
>>>> 
>>>> also, when i run my jar, i provide -
>>>> "-Dmapred.map.child.java.opts="-Xmx4000m" at the end.
>>>> inspite of this, the task is not getting the max heap size im setting.
>>>> 
>>>> where did i go wrong?
>>>> 
>>>> after changing mapred-site.xml, i restarted jobtracker and tasktracker..
is
>>>> that not good enough?
>>>> 
>>>> thanks
>> 
> 
> 


Mime
View raw message