kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangdp <apache_...@163.com>
Subject it run error when run step 1. Execution failed with exit status: 3
Date Sun, 01 Jul 2018 03:07:43 GMT
Hi,

HDP2.3+Kylin 2.2

 when run a bigdata set,hadoop it could't work,add  set hive.auto.convert.join=false can workarounds
for this bug:

Where can I add this command in kylin?

Error log:

Create and distribute table, cmd: 
hive -e "USE default;

DROP TABLE IF EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897
(
PSN_PUR_LOAN_DATE_CD date
,PSN_PUR_LOAN_ORG_CD string
,PSN_PUR_LOAN_CURR_CD string
,PSN_PUR_LOAN_QUAL_CD string
,PSN_PUR_LOAN_LON_BUSI_CD string
,PSN_PUR_LOAN_GBENT_SCALE_CD string
,PSN_PUR_LOAN_IND_INV_CD string
,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9)
,PSN_PUR_LOAN_LON_AMT decimal(25,5)
,PSN_PUR_LOAN_GRNT_AMT decimal(25,5)
,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5)
,PSN_PUR_LOAN_LON_RATE decimal(10,3)
,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3)
,PSN_PUR_LOAN_RECA_PRCP decimal(20,5)
,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5)
,PSN_PUR_LOAN_RECA_INT decimal(20,5)
,PSN_PUR_LOAN_OVDUE_INT decimal(20,5)
,PSN_PUR_LOAN_STOT_DEINT decimal(20,5)
,PSN_PUR_LOAN_LON_BAL decimal(25,5)
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8-7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897';
INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_loan_20d88469_c478_41c6_a480_42c61aa0f897
SELECT
PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD
,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD
,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD
,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD
,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD
,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD
,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD
,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES
,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT
,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT
,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT
,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE
,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE
,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP
,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP
,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT
,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT
,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT
,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL
FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN 
INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL
ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD
INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI
ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD
INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE
ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD
INNER JOIN TEST.DIM_CURR as DIM_CURR
ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD
INNER JOIN TEST.DIM_DATE as DIM_DATE
ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD
INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV
ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD
INNER JOIN TEST.DIM_ORG as DIM_ORG
ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD
WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < '20180201')
;

" --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf
dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true
--hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties
OK
Time taken: 2.163 seconds
OK
Time taken: 0.53 seconds
OK
Time taken: 0.398 seconds
Query ID = hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5
Total jobs = 5
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Execution log at: /tmp/hdfs/hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5.log
2018-07-01 10:46:26 Starting to launch local task to process map join; maximum memory = 1046478848
2018-07-01 10:46:33 Processing rows: 200000 Hashtable size: 199999 Memory usage: 209462696
percentage: 0.20
2018-07-01 10:46:35 Processing rows: 300000 Hashtable size: 299999 Memory usage: 290444464
percentage: 0.278
2018-07-01 10:46:40 Processing rows: 400000 Hashtable size: 399999 Memory usage: 387538096
percentage: 0.37
2018-07-01 10:46:41 Processing rows: 500000 Hashtable size: 499999 Memory usage: 486199336
percentage: 0.465
2018-07-01 10:46:42 Processing rows: 600000 Hashtable size: 599999 Memory usage: 580640608
percentage: 0.555
2018-07-01 10:46:47 Processing rows: 700000 Hashtable size: 699999 Memory usage: 657217728
percentage: 0.628
2018-07-01 10:46:49 Processing rows: 800000 Hashtable size: 799999 Memory usage: 751664416
percentage: 0.718
2018-07-01 10:46:50 Processing rows: 900000 Hashtable size: 899999 Memory usage: 828289456
percentage: 0.792
2018-07-01 10:46:57 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 908630072
percentage: 0.868
2018-07-01 10:47:11 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 994774944
percentage: 0.951
Execution failed with exit status: 3
Obtaining error information

Task failed!
Task ID:
  Stage-34

Logs:

/tmp/hdfs/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask




apache_dev@163.com
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message