Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A958D1171A for ; Wed, 13 Aug 2014 17:44:58 +0000 (UTC) Received: (qmail 54394 invoked by uid 500); 13 Aug 2014 17:44:57 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 54326 invoked by uid 500); 13 Aug 2014 17:44:57 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 54316 invoked by uid 99); 13 Aug 2014 17:44:57 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 17:44:57 +0000 Received: from [IPv6:::1] (localhost [127.0.0.1]) by minotaur.apache.org (Postfix) with ESMTP id 1C31D11719; Wed, 13 Aug 2014 17:44:53 +0000 (UTC) Message-ID: <53EBA413.70706@apache.org> Date: Wed, 13 Aug 2014 10:44:51 -0700 From: Gopal V User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 MIME-Version: 1.0 To: karthik Srivasthava , user@hive.apache.org Subject: Re: New to TEZ References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 8/11/14, 1:48 PM, karthik Srivasthava wrote: Hi, > Below was my log.. I couldnt find where the error is. Can you please point > out what caused my error... The correct items to post back would be hive.tez.container.size, hive.tez.java.opts and the value of io.sort.mb. It does look like you have a small container/wrong -Xmx and a very big sort buffer. Cheers, Gopal > Map 1: -/- Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > Map 1: 0/138 Reducer 2: 0/20 > > > > Status: Failed > Vertex failed, vertexName=Map 1, vertexId=vertex_1407300075548_0189_1_01, > diagno > stics=[Task failed, taskId=task_1407300075548_0189_1_01_000001, > diagnostics=[Att > emptID:attempt_1407300075548_0189_1_01_000001_0 Info:Error: > java.lang.RuntimeExc > eption: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :174) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro > cessorRuntimeTask.java:307) > at > org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5 > 64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma > tion.java:1594) > at > org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55 > 3) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter. nit>(DefaultSorter.java:137) > at > org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile > SortedOutput.java:103) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP > rocessor.java:86) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :164) > ... 6 more > , AttemptID:attempt_1407300075548_0189_1_01_000001_1 Info:Error: > java.lang.Runti > meException: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :174) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro > cessorRuntimeTask.java:307) > at > org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5 > 64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma > tion.java:1594) > at > org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55 > 3) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter. nit>(DefaultSorter.java:137) > at > org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile > SortedOutput.java:103) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP > rocessor.java:86) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :164) > ... 6 more > > Container released by application, > AttemptID:attempt_1407300075548_0189_1_01_000 > 001_2 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError: > Java h > eap space > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :174) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro > cessorRuntimeTask.java:307) > at > org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5 > 64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma > tion.java:1594) > at > org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55 > 3) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter. nit>(DefaultSorter.java:137) > at > org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile > SortedOutput.java:103) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP > rocessor.java:86) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :164) > ... 6 more > > Container released by application, > AttemptID:attempt_1407300075548_0189_1_01_000 > 001_3 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError: > Java h > eap space > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :174) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro > cessorRuntimeTask.java:307) > at > org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5 > 64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma > tion.java:1594) > at > org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55 > 3) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter. nit>(DefaultSorter.java:137) > at > org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile > SortedOutput.java:103) > at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP > rocessor.java:86) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java > :164) > ... 6 more > ], Vertex failed as one or more tasks failed. failedTasks:1] > Vertex killed, vertexName=Reducer 2, > vertexId=vertex_1407300075548_0189_1_00, di > agnostics=[Vertex received Kill while in RUNNING state., Vertex killed as > other > vertex failed. failedTasks:0] > DAG failed due to vertex failure. failedVertices:1 killedVertices:1 > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.tez.T > ezTask > >