hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal V <gop...@apache.org>
Subject Re: New to TEZ
Date Wed, 13 Aug 2014 17:44:51 GMT
On 8/11/14, 1:48 PM, karthik Srivasthava wrote:
Hi,

> Below was my log.. I couldnt find where the error is. Can you please point
> out what caused my error...

The correct items to post back would be hive.tez.container.size, 
hive.tez.java.opts and the value of io.sort.mb.

It does look like you have a small container/wrong -Xmx and a very big 
sort buffer.

Cheers,
Gopal

> Map 1: -/-      Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
> Map 1: 0/138    Reducer 2: 0/20
>
>
>
> Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1407300075548_0189_1_01,
> diagno
> stics=[Task failed, taskId=task_1407300075548_0189_1_01_000001,
> diagnostics=[Att
> emptID:attempt_1407300075548_0189_1_01_000001_0 Info:Error:
> java.lang.RuntimeExc
> eption: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :174)
>          at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
> cessorRuntimeTask.java:307)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
> 64)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:415)
>          at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1594)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
> 3)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
> nit>(DefaultSorter.java:137)
>          at
> org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
> SortedOutput.java:103)
>          at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
> rocessor.java:86)
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :164)
>          ... 6 more
> , AttemptID:attempt_1407300075548_0189_1_01_000001_1 Info:Error:
> java.lang.Runti
> meException: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :174)
>          at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
> cessorRuntimeTask.java:307)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
> 64)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:415)
>          at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1594)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
> 3)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
> nit>(DefaultSorter.java:137)
>          at
> org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
> SortedOutput.java:103)
>          at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
> rocessor.java:86)
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :164)
>          ... 6 more
>
> Container released by application,
> AttemptID:attempt_1407300075548_0189_1_01_000
> 001_2 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError:
> Java h
> eap space
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :174)
>          at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
> cessorRuntimeTask.java:307)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
> 64)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:415)
>          at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1594)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
> 3)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
> nit>(DefaultSorter.java:137)
>          at
> org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
> SortedOutput.java:103)
>          at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
> rocessor.java:86)
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :164)
>          ... 6 more
>
> Container released by application,
> AttemptID:attempt_1407300075548_0189_1_01_000
> 001_3 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError:
> Java h
> eap space
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :174)
>          at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
> cessorRuntimeTask.java:307)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
> 64)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:415)
>          at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
> tion.java:1594)
>          at
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
> 3)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>          at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
> nit>(DefaultSorter.java:137)
>          at
> org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
> SortedOutput.java:103)
>          at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
> rocessor.java:86)
>          at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
> :164)
>          ... 6 more
> ], Vertex failed as one or more tasks failed. failedTasks:1]
> Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1407300075548_0189_1_00, di
> agnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
> other
> vertex failed. failedTasks:0]
> DAG failed due to vertex failure. failedVertices:1 killedVertices:1
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.T
> ezTask
>
>


Mime
View raw message