hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject RE: Hive query hangs in reduce steps
Date Sat, 09 Jan 2016 13:34:36 GMT
Hi Suresh,

 

I have the same issue when I use Hive on Spark.

 

What normally works is Hive on MR. Have you tried:

 

set hive.execution.engine=mr;

 

Sounds like it times out for one reason or other!

 

From: Suresh V [mailto:verditer@gmail.com] 
Sent: 09 January 2016 11:35
To: user@hive.apache.org
Subject: Hive query hangs in reduce steps

 

Dear all

We have a Hive query that 'insert overwrites' from one main hive table to another table about
24million rows every day.

This query was working fine so long, but lately it has started to hang at the reduce steps.


It just gets stuck after all maps are completed. We checked the logs and it says the containers
are released.

The below exception starts to show up in the logs once the reduce steps start., and keeps
recurring.

The job completes fine if we reduce the # of rows processed by reducing the # of days data
being processed. 

 

2016-01-08 19:33:33,091 INFO [IPC Server handler 28 on 43451] org.apache.tez.dag.app.dag.impl.TaskImpl:
TaskAttempt:attempt_1442077641322_71853_1_06_000001_0 sent events: (682-684)
2016-01-08 19:33:33,119 INFO [Socket Reader #1 for port 43451] org.apache.hadoop.ipc.Server:
Socket Reader #1 for port 43451: readAndProcess from client 39.0.8.17 threw exception [java.io.IOException:
Connection reset by peer]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.hadoop.ipc.Server.channelRead(Server.java:2558)
        at org.apache.hadoop.ipc.Server.access$2800(Server.java:130)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1459)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:750)
        at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:624)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595)
2016-01-08 19:33:33,124 INFO [IPC Server handler 24 on 43451] org.apache.tez.dag.app.dag.impl.TaskImpl:
TaskAttempt:attempt_1442077641322_71853_1_06_000000_0 sent events: (682-684)
2016-01-08 19:33:33,125 INFO [AsyncDispatcher event handler] org.apache.tez.dag.history.HistoryEventHandler:
[HISTORY][DAG:dag_1442077641322_71853_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 4, taskAttemptId=attempt_1442077641322_71853_1_07_000750_0,
startTime=1452303118171, finishTime=1452303213123, timeTaken=94952, status=SUCCEEDED, diagnostics=,
counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File
System Counters, FILE: BYTES_READ=56, FILE: BYTES_WRITTEN=554004, FILE: READ_OPS=0, FILE:
LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS: BYTES_READ=503489499, HDFS: BYTES_WRITTEN=0, HDFS:
READ_OPS=29, HDFS: LARGE_READ_OPS=0, HDFS: WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter,
SPILLED_RECORDS=6593, GC_TIME_MILLIS=317, CPU_MILLISECONDS=-765630, PHYSICAL_MEMORY_BYTES=684494848,
VIRTUAL_MEMORY_BYTES=1374089216, COMMITTED_HEAP_BYTES=801112064, INPUT_RECORDS_PROCESSED=9068754,
OUTPUT_RECORDS=6593, OUTPUT_BYTES=540750, OUTPUT_BYTES_WITH_OVERHEAD=553940, OUTPUT_BYTES_PHYSICAL=553948,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0,
org.apache.hadoop.hive.ql.exec.FilterOperator$Counter, FILTERED=28489474, PASSED=22646, org.apache.hadoop.hive.ql.exec.MapOperator$Counter,
DESERIALIZE_ERRORS=0
2

 

Please let me know if more details are required. Request help with this issue.

Thank you

Suresh.


Mime
View raw message