kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maoxiaomao <lang--lang--l...@163.com>
Subject Re:What determine the file num When MERGE Step. Convert Cuboid Data to HFile
Date Fri, 16 Aug 2019 10:09:38 GMT
Inaddion I'm useing:
 kylin2.6.1
 hbase 2.0.0-cdh6.0.0
 add for this step there is 1500Mappers,100reduers. 
 50 region for the hbase Table
 There is some log when reducer writing file : (It wrote about 900M for the first file,but
only less then 100Bfor anoter files)
2019-08-16 14:42:46,805 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30000001
2019-08-16 14:42:46,805 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 8604m
2019-08-16 14:42:47,529 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30100001
2019-08-16 14:42:47,529 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 8465m
2019-08-16 14:42:48,821 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30200001
2019-08-16 14:42:48,821 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 8256m
2019-08-16 14:42:49,875 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30300001
2019-08-16 14:42:49,875 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 8118m
2019-08-16 14:42:50,403 WARN [DataStreamer for file /data/kylin/kylin_metadata/kylin-6bc4c609-25d6-8628-0123-7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F2/72469283b65f478c835bbf774180ec0a]
org.apache.hadoop.hdfs.DataStreamer: These favored nodes were specified but not chosen: [c3-c03-136-30-16.yidian.com:16020]
Specified favored nodes: [c3-c03-136-30-16.yidian.com:16020]
2019-08-16 14:42:50,699 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30400001
2019-08-16 14:42:50,699 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 7930m
2019-08-16 14:42:51,798 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer
Key with ordinal: 30500001
2019-08-16 14:42:51,798 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available
memory: 7699m
2019-08-16 14:42:51,988 INFO [main] org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2:
Writer=hdfs://yellow/data/kylin/kylin_metadata/kylin-6bc4c609-25d6-8628-0123-7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/06f3b0e9f0644ee782b7cf4469f44a70,
wrote=893827310
7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/1454ea148f1547499209a266ad25387f,
wrote=61
7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/9d35446634154b4ca4be56f361b57c8b,
wrote=55

At 2019-08-16 15:58:23, "maoxiaomao" <lang--lang--lang@163.com> wrote:

Hi,
       I excuted a MERGE job, in Step. Convert Cuboid Data to HFile ,there is about 30,000,000
files in path /kylin_metadata/kylin-7645ff8a-8fa4-36f7-c052-a0b18b1526b7/{cubeName}/hfile/_temporary/1/_temporary/attempt_1558444096078_504979_r_0000**_1/F1
, file name looks like in increasing sort (pic 1.).
I wonder this file num determined by what factors?Hoping for reply :) 


pic 1.




 
Mime
View raw message