kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "240902713" <240902...@qq.com>
Subject 回复:Kylin Build get stuck at Build Dimension Dictionary
Date Thu, 11 Jul 2019 13:24:52 GMT
Hi Jason Tang , 


Have you tried to increase the memory of kylin from 16g to 32g? 
Build Dimension Dictionary is executed in memory .


------------------ 原始邮件 ------------------
发件人: "Jason Tang"<jason.tang@striking.ly>;
发送时间: 2019年7月8日(星期一) 中午1:37
收件人: "user"<user@kylin.apache.org>;

主题: Kylin Build get stuck at Build Dimension Dictionary



Kylin cube build jod get stuck at “Build Dimension Dictionary” step. No error is shown
in the log, as below:



2019-07-08 03:55:37,374 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:280 : Done broadcasting
UPDATE, cube, sxl_pageviews_clone
2019-07-08 03:55:37,432 INFO  [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
steps.CreateDictionaryJob:96 : Dict for 'IP_ADDRESS' not pre-built.
2019-07-08 03:55:37,432 DEBUG [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
cli.DictionaryGeneratorCLI:76 : Dict for 'IP_ADDRESS' not pre-built, build it from hdfs://ip-10-201-1-15.cn-north-1.compute.internal:8020/kylin/kylin_metadata/kylin-a849e172-ecd0-6438-bca3-538e151a4594/sxl_pageviews_clone/fact_distinct_columns/SXL_PAGEVIEWS_PARQUET.IP_ADDRESS
2019-07-08 03:55:37,482 INFO  [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
dict.DictionaryManager:276 : building dictionary for DEFAULT.SXL_PAGEVIEWS_PARQUET.IP_ADDRESS
2019-07-08 03:55:37,538 INFO  [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
dict.DictionaryManager:285 : Building dictionary object {"uuid":"459ff7cf-ceb3-b751-8b03-f7d77b012d22","last_modified":0,"version":"2.6.1.0","source_table":"DEFAULT.SXL_PAGEVIEWS_PARQUET","source_column":"IP_ADDRESS","source_column_index":5,"data_type":"varchar(256)","input":{"path":"hdfs://ip-10-201-1-15.cn-north-1.compute.internal:8020/kylin/kylin_metadata/kylin-a849e172-ecd0-6438-bca3-538e151a4594/sxl_pageviews_clone/fact_distinct_columns/SXL_PAGEVIEWS_PARQUET.IP_ADDRESS","size":1180625,"last_modified_time":1562557443756},"dictionary_class":null,"cardinality":0}
2019-07-08 03:55:37,721 INFO  [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
mr.DFSFileTableReader:197 : Auto detect delim to be null, will take THE-WHOLE-LINE as a single
value, for hdfs://ip-10-201-1-15.cn-north-1.compute.internal:8020/kylin/kylin_metadata/kylin-a849e172-ecd0-6438-bca3-538e151a4594/sxl_pageviews_clone/fact_distinct_columns/SXL_PAGEVIEWS_PARQUET.IP_ADDRESS/-r-00008
2019-07-08 03:55:37,724 DEBUG [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
util.ZookeeperDistributedLock:147 : 103-12468@f768cdc068ec trying to lock /kylin/kylin_metadata/dict/DEFAULT.SXL_PAGEVIEWS_PARQUET_IP_ADDRESS/lock
2019-07-08 03:55:37,727 INFO  [Scheduler 132467998 Job a849e172-ecd0-6438-bca3-538e151a4594-103]
util.ZookeeperDistributedLock:159 : 103-12468@f768cdc068ec acquired lock at /kylin/kylin_metadata/dict/DEFAULT.SXL_PAGEVIEWS_PARQUET_IP_ADDRESS/lock
2019-07-08 03:56:02,567 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:56:14,838 DEBUG [http-bio-7070-exec-6] common.KylinConfig:328 : KYLIN_CONF property
was not set, will seek KYLIN_HOME env variable
2019-07-08 03:56:14,838 INFO  [http-bio-7070-exec-6] common.KylinConfig:334 : Use KYLIN_HOME=/hadoop/install/kylin
2019-07-08 03:56:17,625 DEBUG [http-bio-7070-exec-9] security.KylinAuthenticationProvider:114
: User ADMIN authorities : [ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER]
2019-07-08 03:56:17,632 INFO  [http-bio-7070-exec-9] common.KylinConfig:457 : Creating new
manager instance of class org.apache.kylin.rest.security.KylinUserManager
2019-07-08 03:56:17,632 INFO  [http-bio-7070-exec-9] security.KylinUserManager:61 : Initializing
KylinUserManager with config kylin_metadata@hbase
2019-07-08 03:56:17,633 DEBUG [http-bio-7070-exec-9] cachesync.CachedCrudAssist:118 : Reloading
ManagedUser from kylin_metadata(key='/user')@kylin_metadata@hbase
2019-07-08 03:56:17,674 DEBUG [http-bio-7070-exec-9] cachesync.CachedCrudAssist:127 : Loaded
1 ManagedUser(s) out of 1 resource
2019-07-08 03:56:17,677 DEBUG [http-bio-7070-exec-9] security.KylinAuthenticationProvider:128
: Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@c8fb3bba:
Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED];
Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@0:
RemoteIpAddress: 10.201.101.229; SessionId: 6CCE82A0150C881F5EE0CEE7E9EFD509; Granted Authorities:
ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-08 03:56:18,000 DEBUG [http-bio-7070-exec-9] controller.UserController:52 : User login:
org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED];
Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true;
Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2019-07-08 03:56:19,379 DEBUG [http-bio-7070-exec-1] project.ProjectL2Cache:198 : Loading
L2 project cache for user_analytics_sxl
2019-07-08 03:56:19,426 INFO  [http-bio-7070-exec-1] common.KylinConfig:457 : Creating new
manager instance of class org.apache.kylin.metadata.realization.RealizationRegistry
2019-07-08 03:56:19,427 INFO  [http-bio-7070-exec-1] realization.RealizationRegistry:54 :
Initializing RealizationRegistry with metadata url kylin_metadata@hbase
2019-07-08 03:56:19,429 INFO  [http-bio-7070-exec-1] common.KylinConfig:457 : Creating new
manager instance of class org.apache.kylin.storage.hybrid.HybridManager
2019-07-08 03:56:19,442 INFO  [http-bio-7070-exec-1] hybrid.HybridManager:71 : Initializing
HybridManager with config kylin_metadata@hbase
2019-07-08 03:56:19,443 DEBUG [http-bio-7070-exec-1] cachesync.CachedCrudAssist:118 : Reloading
HybridInstance from kylin_metadata(key='/hybrid')@kylin_metadata@hbase
2019-07-08 03:56:19,454 DEBUG [http-bio-7070-exec-1] cachesync.CachedCrudAssist:127 : Loaded
0 HybridInstance(s) out of 0 resource
2019-07-08 03:56:19,455 INFO  [http-bio-7070-exec-1] realization.RealizationRegistry:81 :
RealizationRegistry is {CUBE=org.apache.kylin.cube.CubeManager@799f55ea, HYBRID=org.apache.kylin.storage.hybrid.HybridManager@1fb18590}
2019-07-08 03:56:25,507 DEBUG [http-bio-7070-exec-4] badquery.BadQueryHistoryManager:65 :
Loaded 50 Bad Query(s)
2019-07-08 03:56:30,150 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 03:56:32,689 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:57:02,522 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:57:30,231 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 03:57:32,709 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:58:02,521 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:58:30,231 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 03:58:32,525 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:59:02,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 03:59:20,846 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-74.cn-north-1.compute.internal/10.201.1.74:16000
2019-07-08 03:59:20,847 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-74.cn-north-1.compute.internal/10.201.1.74:16000
2019-07-08 03:59:20,847 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-245.cn-north-1.compute.internal/10.201.1.245:16020
2019-07-08 03:59:20,847 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-201.cn-north-1.compute.internal/10.201.1.201:16020
2019-07-08 03:59:30,231 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 03:59:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:00:02,521 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:00:30,232 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:00:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:01:02,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:01:30,272 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:01:32,844 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:02:02,695 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:02:30,273 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:02:32,775 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:03:02,648 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:03:30,413 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:03:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:04:02,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:04:30,413 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:04:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:05:02,862 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:05:30,414 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:05:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:06:02,922 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:06:30,414 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:06:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:07:02,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:07:20,937 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-245.cn-north-1.compute.internal/10.201.1.245:16020
2019-07-08 04:07:20,938 INFO  [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 :
Cleanup idle connection to ip-10-201-1-201.cn-north-1.compute.internal/10.201.1.201:16020
2019-07-08 04:07:30,496 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.
2019-07-08 04:07:32,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:08:00,503 DEBUG [http-bio-7070-exec-8] badquery.BadQueryHistoryManager:65 :
Loaded 50 Bad Query(s)
2019-07-08 04:08:02,520 INFO  [FetcherRunner 318803806-95] threadpool.DefaultFetcherRunner:94
: Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 1939 already succeed,
0 error, 2 discarded, 0 others
2019-07-08 04:08:30,695 INFO  [BadQueryDetector] service.BadQueryDetector:147 : Detect bad
query.












The amount of data is not large(as shown in the picture). Any suggestions about this situation?


Best Regards
Jason Tang
Mime
View raw message