hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: rack awareness in hadoop
Date Sun, 21 Apr 2013 00:19:16 GMT
And don't forget to look at unlimit settings as well

Sent from my iPhone

On Apr 20, 2013, at 5:07 PM, Marcos Luis Ortiz Valmaseda <marcosluis2186@gmail.com>
wrote:

> Like, Aaron say, this problem is related the Linux memory manager.
> You can tune it using the vm.overcommit_memory=1.
> Before to do any change, read all resources first:
> http://www.thegeekstuff.com/2012/02/linux-memory-swap-cache-shared-vm/
> http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html
> http://lwn.net/Articles/317814/
> http://www.tldp.org/LDP/tlk/mm/memory.html
> 
> To learn more about how to tune kernel variables for Hadoop applications. Read these
links too:
> First, the amazing Hadoop OperationsĀ“s book from Eric:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2685120
> 
> Hadoop Performance Tuning Guide from AMD:
> http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf
> 
> IntelĀ® Distribution for Apache Hadoop*  Software: Optimization and Tuning Guide:
> http://hadoop.intel.com/pdfs/IntelDistributionTuningGuide.pdf
> 
> Best wishes.
> 
> 
> 
> 2013/4/20 Aaron Eng <aeng@maprtech.com>
>> The problem is probably not related to the JVM memory so much as the Linux memory
manager.  The exception is in java.lang.UNIXProcess.<init>(UNIXProcess.java:148) which
would imply this is happening when trying to create a new process.  The initial malloc for
the new process space is being denied by the memory manager.  There could be many reasons
why this happens, though the most likely is your overcommit settings and swap space.  I'd
suggest reading through these details:
>> 
>> https://www.kernel.org/doc/Documentation/vm/overcommit-accounting
>> 
>> On Sat, Apr 20, 2013 at 4:00 PM, Kishore Yellamraju <kishore@rocketfuelinc.com>
wrote:
>>> All,
>>> 
>>> I have posted this question to CDH ML ,  but i guess i can post it here because
its a general hadoop question.
>>> 
>>> When the NN or JT gets the rack info, i guess it stores the info in memory. can
i ask you where in the JVM memory it will store the results ( perm gen ?) ? .  I am getting
"cannot allocate memory on NN and JT " and they have more than enough memory. when i looked
at JVM usage stats i can see it doesnt have enough perm free space.so if its storing the values
in perm gen  then there is a chance of this memory issues.
>>> 
>>> 
>>> Thanks in advance !!!
>>> 
>>> 
>>> exception that i see in logs :
>>> 
>>> java.io.IOException: Cannot run program "/etc/hadoop/conf/topo.sh" (in directory
"/usr/lib/hadoop-0.20-mapreduce"): java.io.IOException: error=12, Cannot allocate memory
>>>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>>         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>>         at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>>         at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>>>         at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
>>>         at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>>>         at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>>>         at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:593)
>>>         at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:765)
>>>         at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)
>>>         at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>         at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate
memory
>>>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>>         at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>>         ... 14 more
>>> 2013-04-20 02:07:28,298 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization
failed:
>>> java.lang.NullPointerException
>>> 
>>> 
>>> -Thanks
>>>  kishore kumar yellamraju |Ground control operations|kishore@rocketfuel.com |
408.203.0424
>>> 
>>> 
> 
> 
> 
> -- 
> Marcos Ortiz Valmaseda,
> Data-Driven Product Manager at PDVSA
> Blog: http://dataddict.wordpress.com/
> LinkedIn: http://www.linkedin.com/in/marcosluis2186
> Twitter: @marcosluis2186

Mime
View raw message