kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jyzheng" <jyzh...@iflytek.com>
Subject 答复: kylin query failed
Date Tue, 17 May 2016 02:16:51 GMT
1.     The cardinality of app_name is about 30,000. 

2.     Yes, the Kylin has crashed after query failed.

3.     The clue of kylin.log in the given below:

 

Thanks for attention!

 

 

2016-05-17 10:13:51,531 INFO  [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher:
0 running, 0 actual running, 0 ready, 17 others

2016-05-17 10:14:20,031 DEBUG [http-bio-7070-exec-5] service.AdminService:90 : Get Kylin Runtime
Config

2016-05-17 10:14:20,037 DEBUG [http-bio-7070-exec-8] controller.UserController:64 : authentication.getPrincipal()
is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER

2016-05-17 10:14:20,321 DEBUG [http-bio-7070-exec-5] controller.ProjectController:97 : authentication.getPrincipal()
is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER

2016-05-17 10:14:25,467 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal()
is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER

2016-05-17 10:14:25,480 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal()
is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER

2016-05-17 10:14:25,610 INFO  [http-bio-7070-exec-1] controller.TableController:89 : Return
all table metadata in 1 seconds

2016-05-17 10:14:28,341 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal()
is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password:
[PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked:
true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER

2016-05-17 10:14:28,570 DEBUG [http-bio-7070-exec-5] service.QueryService:293 : getting table
metas

2016-05-17 10:14:28,574 DEBUG [http-bio-7070-exec-5] service.QueryService:311 : getting column
metas

2016-05-17 10:14:28,601 DEBUG [http-bio-7070-exec-5] service.QueryService:325 : done column
metas

2016-05-17 10:14:51,542 INFO  [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher:
0 running, 0 actual running, 0 ready, 17 others

2016-05-17 10:14:52,457 INFO  [http-bio-7070-exec-10] controller.QueryController:175 : Using
project: compass

2016-05-17 10:14:52,458 INFO  [http-bio-7070-exec-10] controller.QueryController:176 : The
original query:  select app_name, count(distinct uid) as uv

 

from sdk_log

 

left join week_calendar on sdk_log.day_time = week_calendar.week_cal

 

where sdk_log.day_time = date '2016-05-08'

 

group by app_name

 

order by uv desc

 

limit 200 

 

 

2016-05-17 10:14:52,471 INFO  [http-bio-7070-exec-10] service.QueryService:269 : The corrected
query: select app_name, count(distinct uid) as uv

 

from sdk_log

 

left join week_calendar on sdk_log.day_time = week_calendar.week_cal

 

where sdk_log.day_time = date '2016-05-08'

 

group by app_name

 

order by uv desc

 

limit 200

2016-05-17 10:14:53,938 INFO  [http-bio-7070-exec-10] routing.QueryRouter:48 : The project
manager's reference is org.apache.kylin.metadata.project.ProjectManager@58baceac

2016-05-17 10:14:53,941 INFO  [http-bio-7070-exec-10] routing.QueryRouter:60 : Find candidates
by table DEFAULT.SDK_LOG and project=COMPASS : org.apache.kylin.query.routing.Candidate@70f24f62

2016-05-17 10:14:53,947 INFO  [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule:
class org.apache.kylin.query.routing.rules.RemoveUncapableRealizationsRule, realizations before:
[sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]

2016-05-17 10:14:53,948 INFO  [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule:
class org.apache.kylin.query.routing.rules.RealizationSortRule, realizations before: [sdk_cube(CUBE)],
realizations after: [sdk_cube(CUBE)]

2016-05-17 10:14:53,948 INFO  [http-bio-7070-exec-10] routing.QueryRouter:72 : The realizations
remaining: [sdk_cube(CUBE)] And the final chosen one is the first one

2016-05-17 10:14:54,662 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:107 : query
storage...

2016-05-17 10:14:54,749 INFO  [http-bio-7070-exec-10] cache.AbstractCacheFledgedQuery:85 :
Cache for 95f6d938-b974-4a92-b858-c0522faa7ff4 initializing...

2016-05-17 10:14:54,797 INFO  [http-bio-7070-exec-10] cache.CacheFledgedStaticQuery:58 : no
existing cache to use

2016-05-17 10:14:54,802 INFO  [http-bio-7070-exec-10] v2.CubeStorageQuery:252 : exactAggregation
is true

2016-05-17 10:14:54,812 INFO  [http-bio-7070-exec-10] v2.CubeStorageQuery:358 : Memory budget
is set to: 49146

2016-05-17 10:14:54,834 INFO  [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583)
loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/APP_NAME/21980b9c-a872-4fb6-aef4-3a96ed2ef1c8.dict

2016-05-17 10:14:54,959 INFO  [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583)
loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/DAY_TIME/4b6b9a89-8666-440f-94ce-cf3b2d68fc44.dict

2016-05-17 10:14:55,037 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:248 : New scanner
for current segment sdk_cube[20160502000000_20170103000000] will use SCAN_FILTER_AGGR_CHECKMEM
as endpoint's behavior

2016-05-17 10:14:55,074 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:283 : Serialized
scanRequestBytes 216 bytes, rawScanBytesString 81 bytes

2016-05-17 10:14:55,075 INFO  [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:286 : The scan(s)
info for current segment is as below, shard part of start/end key is set to 0

2016-05-17 10:14:55,082 INFO  [http-bio-7070-exec-10] v2.CubeHBaseRPC:309 : Visiting hbase
table KYLIN_F6NPB6IA0Z: cuboid exact match, from 5 to 5 Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB
(\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B<\xCB) Stop:  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B\x3C\xCB\x00
(\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B<\xCB\x00) Fuzzy key counts: 1.
Fuzzy keys : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00;

2016-05-17 10:14:55,083 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:292 : Submitting
rpc to 2 shards starting from shard 1, scan requests count 1

2016-05-17 10:14:55,172 INFO  [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:123 : Timeout
for ExpectedSizeIterator is 60000

2016-05-17 10:14:55,189 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:127 : return
TupleIterator...

2016-05-17 10:14:55,190 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:128 : Storage
cache used for this storage query:null

2016-05-17 10:15:05,367 INFO  [pool-6-thread-2] v2.CubeHBaseEndpointRPC:320 : <spawned
by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x32\x32\x37\x39\x34\x66\x30\x37\x39\x34\x37\x63\x39\x34\x36\x30\x34\x64\x33\x66\x66\x36\x33\x34\x31\x62\x36\x34\x65\x34\x30\x33\x2E
on host: hfa-pro0043.hadoop.cpcc.iflyyun.cn.Total scanned row: 128455. Total filtered/aggred
row: 0. Time elapsed in EP: 9379(ms). Server CPU usage: 0.6528155468594548, server physical
mem left: 1.5867633664E10, server swap mem left:1.00663287808E11.Etc message: 1,1183,9378,9379,.

2016-05-17 10:15:06,390 DEBUG [pool-6-thread-2] util.CompressionUtils:65 : Original: 36540710
bytes. Decompressed: 54563500 bytes

2016-05-17 10:15:06,401 INFO  [http-bio-7070-exec-10] lookup.SnapshotManager:178 : Loading
snapshotTable from /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot,
with loadData: true

2016-05-17 10:15:06,436 DEBUG [http-bio-7070-exec-10] lookup.SnapshotManager:184 : Loaded
snapshot at /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot

2016-05-17 10:15:08,057 INFO  [pool-6-thread-1] v2.CubeHBaseEndpointRPC:320 : <spawned
by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x00\x01\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x37\x65\x39\x63\x38\x30\x37\x33\x31\x62\x33\x34\x63\x62\x63\x32\x36\x35\x33\x34\x65\x32\x39\x64\x65\x35\x38\x39\x64\x36\x38\x66\x2E
on host: hfa-pro0042.hadoop.cpcc.iflyyun.cn.Total scanned row: 127642. Total filtered/aggred
row: 0. Time elapsed in EP: 11935(ms). Server CPU usage: 0.9497198437438346, server physical
mem left: 1.416450048E9, server swap mem left:1.00663287808E11.Etc message: 1,1278,11935,11935,.

2016-05-17 10:15:09,453 DEBUG [pool-6-thread-1] util.CompressionUtils:65 : Original: 35193219
bytes. Decompressed: 53229962 bytes

 

发件人: user-return-560-jyzheng=iflytek.com@kylin.apache.org [mailto:user-return-560-jyzheng=iflytek.com@kylin.apache.org]
代表 Li Yang
发送时间: 2016年5月15日 19:32
收件人: user@kylin.apache.org
抄送: dev@kylin.apache.org
主题: Re: kylin query failed

 

How high is the cardinality? By saying "failed" and then have to restart, do you mean crash?
Any clue in kylin.log?

Sorry for many questions, but it's hard to help without the details. The new 1.5.2 release
will come with a diagnosis tool that can extract necessary info into a zip, which you can
share with community to diagnose.

 

On Mon, May 9, 2016 at 5:12 PM, jyzheng <jyzheng@iflytek.com <mailto:jyzheng@iflytek.com>
> wrote:

 

When I send sql query to Kylin like this:

 

select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-01'
group by app_name
order by uv desc
limit 200 

 

 

and failed. So I have to restart Kylin. 

 

The dimension `app_name` is a great high cardinality. If I switch to `app_tag` dimension ,
it will return the right result. So I can’t get the top measure dimension in Kylin? If can,
and what can I do to solve this?

--------------------------------

郑江雨  云平台

Phone: 15155195496 <tel:15155195496> 

 

 


Mime
View raw message