hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: Java heap space error
Date Mon, 22 Oct 2012 14:16:11 GMT
Subash,

I have been experiencing this type of an error at some point and no matter how much I played
with heap size  it didn't work. What I found out at the end is I was running out of physical
memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space
with "$hadop fs -df


From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error

I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster
was running until there was an error in the name node log file and I had to reformat it put
the data back

Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research
I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still
have the same issue. It seems to fail near the 100% mapper mark

I checked the log files and the only thing that it does output is java heap space error. Nothing
more.

Any help would be appreciated.

Thanks
Subash

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Mime
View raw message