hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Pivovarov <apivova...@gmail.com>
Subject hive 0.14.1 + tez 0.4 sometimes throws NPE on small data
Date Fri, 09 Jan 2015 19:53:39 GMT
I noticed that if amount of data which should be sent to reducers is small
then tez might throw NPE.

workaround is to set number of reducers to some small value.
e.g. if initially I have
set hive.exec.reducers.max=25;
then to solve the issue I might try
set hive.exec.reducers.max=13;
or even smaller value

I'm using hdp-2.1.5 (which is hive-0.14.1 + tez-0.4.0)

Map 1: 17/21    Map 11: 1/1     Map 14: 4/4     Map 16: 4/4     Map 18:
4/4     Map 19: 1/1     Map 21: 1/1     Map 22: 5/5     Map 23: 18/21   Map
24: 1/1     Map 4: 21/21    Map 6: 1/1      Map 9: 1/1    1
Status: Failed
Vertex re-running, vertexName=Map 6,
vertexId=vertex_1420058390346_15040_1_20
Vertex re-running, vertexName=Map 18,
vertexId=vertex_1420058390346_15040_1_10
Vertex re-running, vertexName=Map 22,
vertexId=vertex_1420058390346_15040_1_14
Vertex re-running, vertexName=Map 4,
vertexId=vertex_1420058390346_15040_1_22
Vertex re-running, vertexName=Map 9,
vertexId=vertex_1420058390346_15040_1_17
Vertex re-running, vertexName=Map 19,
vertexId=vertex_1420058390346_15040_1_02
Vertex re-running, vertexName=Map 24,
vertexId=vertex_1420058390346_15040_1_09
Vertex re-running, vertexName=Map 1,
vertexId=vertex_1420058390346_15040_1_24
Vertex re-running, vertexName=Map 14,
vertexId=vertex_1420058390346_15040_1_15
Vertex re-running, vertexName=Map 16,
vertexId=vertex_1420058390346_15040_1_12
Vertex re-running, vertexName=Map 14,
vertexId=vertex_1420058390346_15040_1_15
Vertex re-running, vertexName=Map 6,
vertexId=vertex_1420058390346_15040_1_20
Vertex re-running, vertexName=Map 18,
vertexId=vertex_1420058390346_15040_1_10
Vertex re-running, vertexName=Map 22,
vertexId=vertex_1420058390346_15040_1_14
Vertex re-running, vertexName=Map 21,
vertexId=vertex_1420058390346_15040_1_19
Vertex re-running, vertexName=Map 16,
vertexId=vertex_1420058390346_15040_1_12
Vertex re-running, vertexName=Map 24,
vertexId=vertex_1420058390346_15040_1_09
Vertex re-running, vertexName=Map 11,
vertexId=vertex_1420058390346_15040_1_06
Vertex failed, vertexName=Reducer 17,
vertexId=vertex_1420058390346_15040_1_11, diagnostics=[Task failed,
taskId=task_1420058390346_15040_1_11_000007,
diagnostics=[AttemptID:attempt_1420058390346_15040_1_11}
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
        at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
        at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
        ... 6 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
13:04:53","reducesinkkey2":"2011-11-0}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
        at
org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
        at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
        ... 8 more

Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_000007_1 Info:Node blacklisted,
AttemptID:attempt_1420058390346_15040_1_11_000007_2 Info:Node blacklisted
Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_000007_3 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: H}
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
        at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
        at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
        ... 6 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
13:04:53","reducesinkkey2":"2011-11-0}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
        at
org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
        at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
        ... 8 more

Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_000007_4 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: H}
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
        at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
        at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
        ... 6 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
13:04:53","reducesinkkey2":"2011-11-0}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
        at
org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
        at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
        ... 8 more

Container released by application,
AttemptID:attempt_1420058390346_15040_1_11_000007_5 Info:Error:
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: H}
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:188)
        at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
        at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:564)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
        at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:553)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02 13:04:53"}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:349)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:228)
        at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:165)
        ... 6 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0)
{"key":{"reducesinkkey0":61419,"reducesinkkey1":"2011-10-02
13:04:53","reducesinkkey2":"2011-11-0}
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:334)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:341)
        at
org.apache.hadoop.hive.ql.exec.PTFOperator.processOp(PTFOperator.java:115)
        at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
        at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:325)
        ... 8 more
], Vertex failed as one or more tasks failed. failedTasks:1]
Vertex killed, vertexName=Reducer 13,
vertexId=vertex_1420058390346_15040_1_03, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 3,
vertexId=vertex_1420058390346_15040_1_00, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 25,
vertexId=vertex_1420058390346_15040_1_08, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 15,
vertexId=vertex_1420058390346_15040_1_13, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 20,
vertexId=vertex_1420058390346_15040_1_01, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 8,
vertexId=vertex_1420058390346_15040_1_07, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 12,
vertexId=vertex_1420058390346_15040_1_04, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Map 23, vertexId=vertex_1420058390346_15040_1_05,
diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
other vertex failed. failedTasks:0]
Vertex killed, vertexName=Map 1, vertexId=vertex_1420058390346_15040_1_24,
diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
other vertex failed. failedTasks:0]
Vertex killed, vertexName=Reducer 5,
vertexId=vertex_1420058390346_15040_1_21, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1420058390346_15040_1_23, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 10,
vertexId=vertex_1420058390346_15040_1_16, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
Vertex killed, vertexName=Reducer 7,
vertexId=vertex_1420058390346_15040_1_18, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0]
DAG failed due to vertex failure. failedVertices:1 killedVertices:13
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message