hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From renguihe <rengu...@ebupt.com>
Subject Hive job leading Hadoop datanodes to dead
Date Wed, 25 Dec 2013 11:29:20 GMT
Hi,
I encountered a problem when I tried to run a Hive(version 0.11.0) job using dynamic partition
on Hadoop(version 1.0.4 and 1.2.1,both) ,the job failed and made some datanodes lost heartbeats
and turned to be dead gradually.
The JDK version is 1.6.0_45/1.6.0_30/1.6.0_43  ,the OS version is Red Hat Enterprise Linux
Server release 6.4 (Santiago) .
I tried the versions all of above,but it still can not work.
 
Here is the Hive script:
 set hive.exec.dynamic.partition.mode=nonstrict;
FROM (SELECT  *
        FROM SRC_IREAD.TEMP_IREAD_LOGON
      ) A
INSERT OVERWRITE TABLE src_iread.src_iread_logon_soft_normal partition(record_day)
SELECT A.* ,
        substr(recordtime,1,8)
INSERT OVERWRITE TABLE src_iread.src_iread_logon_soft partition(record_day)
SELECT A.* ,
        substr(recordtime,1,8)
  ;

Here is the Java error info from the datanode logs:
2013-12-23 00:52:35,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.20.15:50010,
storageID=DS-795828338-192.168.20.15-50010-1387540641089, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:265)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:312)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:376)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:532)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:398)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:107)
at java.lang.Thread.run(Thread.java:662)

Finally,there were tons of logs like this,until manually shutdown the datanode:
2013-12-23 01:08:06,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:07,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:08,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:09,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:10,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:11,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:12,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:13,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:14,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:15,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:16,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:17,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:18,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:19,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:20,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:21,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:22,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:23,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1
2013-12-23 01:08:24,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for
threadgroup to exit, active threads is 1

PS:table DDL:

 CREATE TABLE IF NOT EXISTS TEMP_IREAD_LOGON (
        msisdn string COMMENT '$',
        recordtime string COMMENT '$',
        terminal string COMMENT '$',
        platform string COMMENT '$',
        channelcode string COMMENT '$',
        wapip string COMMENT '$',
        sessionid string COMMENT '$',
        provinceid string COMMENT '$',
        cityid string COMMENT '$',
        clientchannelcode string COMMENT '$',
        access_points string COMMENT '$',
        logintype string COMMENT '$',
        visittype string COMMENT '$',
        sourcetype string COMMENT '$',
        portalip string COMMENT '$' )
 COMMENT 'TMP_IREAD_LOGON INFORMATION OF SRC_IREAD'
 ROW FORMAT DELIMITED
 FIELDS TERMINATED BY '|'
 STORED AS TEXTFILE

 CREATE TABLE IF NOT EXISTS SRC_IREAD_LOGON_SOFT_NORMAL (
        msisdn string COMMENT '',
        recordtime string COMMENT '',
        terminal string COMMENT '$',
        platform int COMMENT '',
        channelcode string COMMENT '',
        wapip string COMMENT '',
        sessionid string COMMENT '',
        provinceid string COMMENT '',
        cityid string COMMENT '',
        clientchannelcode string COMMENT '',
        access_points int COMMENT '1?WLAN 2?CMWAP3?CMNET4?UNKNOWN',
        logintype int COMMENT '???? 1 ??CMWAP???? 2 ?????????? 3 ???????????? 4 ????',
        visittype int COMMENT '1? ???2? ?????????WAP??????????????',
        sourcetype int COMMENT,
        portalip string COMMENT '' )
 COMMENT 'ENTERPRISE INFORMATION OF ODS_IREAD'
 PARTITIONED BY (record_day string)
 ROW FORMAT DELIMITED
 FIELDS TERMINATED BY '|'
 LINES TERMINATED BY '\n'
 STORED AS TEXTFILE

 CREATE TABLE IF NOT EXISTS SRC_IREAD_LOGON_SOFT (
        msisdn string COMMENT '',
        recordtime string COMMENT '',
        terminal string COMMENT '$',
        platform int COMMENT '',
        channelcode string COMMENT '',
        wapip string COMMENT '',
        sessionid string COMMENT '',
        provinceid string COMMENT '',
        cityid string COMMENT '',
        clientchannelcode string COMMENT '',
        access_points int COMMENT '1?WLAN 2?CMWAP3?CMNET4?UNKNOWN',
        logintype int COMMENT '???? 1 ??CMWAP???? 2 ?????????? 3 ???????????? 4 ????',
        visittype int COMMENT '1? ???2? ?????????WAP??????????????',
        sourcetype int ,
        portalip string COMMENT '' )
 COMMENT 'ENTERPRISE INFORMATION OF SRC_IREAD_LOGON_SOFT'
 PARTITIONED BY (record_day string)
 ROW FORMAT DELIMITED
 FIELDS TERMINATED BY '|'
 STORED AS TEXTFILE


could anyone help me out of this problem?
THX!



renguihe
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message