Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EC5CEDD4A for ; Mon, 22 Oct 2012 14:16:48 +0000 (UTC) Received: (qmail 97665 invoked by uid 500); 22 Oct 2012 14:16:44 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 97248 invoked by uid 500); 22 Oct 2012 14:16:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 97230 invoked by uid 99); 22 Oct 2012 14:16:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Oct 2012 14:16:43 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [206.47.135.205] (HELO Spam1.prd.mpac.ca) (206.47.135.205) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Oct 2012 14:16:35 +0000 Received: from Spam1.prd.mpac.ca (unknown [127.0.0.1]) by IMSVA80 (Postfix) with ESMTP id 40C031D8057 for ; Mon, 22 Oct 2012 10:16:12 -0400 (EDT) Received: from SMAIL1.prd.mpac.ca (unknown [172.29.2.53]) by Spam1.prd.mpac.ca (Postfix) with ESMTP id EF7CF1D8049 for ; Mon, 22 Oct 2012 10:16:11 -0400 (EDT) Received: from SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb]) by SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb%16]) with mapi id 14.02.0318.001; Mon, 22 Oct 2012 10:16:11 -0400 From: "Kartashov, Andy" To: "user@hadoop.apache.org" Subject: RE: Java heap space error Thread-Topic: Java heap space error Thread-Index: AQHNr46u5NCkvLdbBUWWvuq/0CQrkpfFX4vA Date: Mon, 22 Oct 2012 14:16:11 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.29.60.102] Content-Type: multipart/alternative; boundary="_000_BD42F346AE90F544A731516A805D1B8AD724D5SMAIL1prdmpacca_" MIME-Version: 1.0 X-TM-AS-Product-Ver: IMSVA-8.0.0.1304-6.5.0.1024-19294.007 X-TM-AS-Result: No--28.038-5.0-31-10 X-imss-scan-details: No--28.038-5.0-31-10 X-TM-AS-Result-Xfilter: Match text exemption rules:No X-TMASE-MatchedRID: gjZGo2H/wj+PrjM/ltMU+dxajlW+zwxCh+w9Wz/xXDr+TXZJvYxoeUsq 2Qq9h9WipvIdHpUEE6EMu/IrMbXELZcw+OXwttp/+8AO9OcJ/m9i7uzdBJI35LV5fSMRD1zqAmL /YMh+r1HM5LpD6JX1RPnxRpaudvBlfF2w37ROyLO+dJWHbg4ITrZrDHFV+TcHbTLvakxk90ZDvi ZnrU/ym97nIKRPK99u+jAPW2r3i0TakVlQZSlZtuYAh37ZsBDCfeQy5LfnzhVVaiL9Jo7jBgn9o fQNoBQFcce8zcwjPUkd28n/9SVHweOuRUn9B+2TMEK/6tg6I3EGF+E6mzeNqAeLCIX046iBjNLx rcxKViUihgJVrNniBWrEBfVjKK6u1lfDCm9+EZVYKMMlFh4BnVqvZZ9/gpIhZ5yuplze9ps57kF jOTI5JYBMc1fifV0YUanNe82h3Oe0q6rdlPIOdCI9MxSOQ6CSLVEjAx3HjK6k7BPGf466/uygQW dkAVMsw6jXr/PvLFu6H+0z6Sb8PDpKoM3F6+9QBxsweNg3EaGFkCkkB0UMNq9dKZJ2VxiaLnuFY faWSqZ4kNKVNaMUV91Vkpneicogfiv3uJ7StNtOvDN+B7cXLhlpFikIFu+YS4W/MRhJ1X4= X-Virus-Checked: Checked by ClamAV on apache.org --_000_BD42F346AE90F544A731516A805D1B8AD724D5SMAIL1prdmpacca_ Content-Type: text/plain; charset="us-ascii" Subash, I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df From: Subash D'Souza [mailto:sdsouza@truecar.com] Sent: Sunday, October 21, 2012 9:19 AM To: user@hadoop.apache.org Subject: Java heap space error I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark I checked the log files and the only thing that it does output is java heap space error. Nothing more. Any help would be appreciated. Thanks Subash NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD724D5SMAIL1prdmpacca_ Content-Type: text/html; charset="us-ascii"

Subash,

 

I have been experiencing this type of an error at some point and no matter how much I played with heap size  it didn’t work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with “$hadop fs –df

 

 

From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error

 

I'm running CDH 4 on  a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back 

 

Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark

 

I checked the log files and the only thing that it does output is java heap space error. Nothing more.

 

Any help would be appreciated.

 

Thanks

Subash

 

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel --_000_BD42F346AE90F544A731516A805D1B8AD724D5SMAIL1prdmpacca_--