kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phong Pham <phongpham1...@gmail.com>
Subject Re: Problem with query cache
Date Wed, 11 Jan 2017 03:11:47 GMT
Hi,
Im using normal query like this:
 select <some dims>, <sum(metrics)> from <table> inner join <table>
where <condition>
group by <some dims>

Result set is about over 10.000 rows
Query time: 3.564

And i have 5 HBase nodes



2017-01-11 9:42 GMT+07:00 Billy Liu <billyliu@apache.org>:

> What kind of query you are using? How much the result set would be? How
> many HBase nodes you are using?
>
> 2017-01-11 10:11 GMT+08:00 Phong Pham <phongpham1805@gmail.com>:
>
>> Hi,
>>    i try to edit config on ehcache_test.xml like this:
>>   <ehcache maxBytesLocalHeap="2048M">
>>
>> And i still not use LDAP authentication.   And its still not working.
>> Result is:
>> *Duration: 3.735*
>> *Project: metrixa_global_database*
>> *Realization Names: [xxx]*
>> *Cuboid Ids: [45971]*
>> *Total scan count: 556952*
>> *Result row count: 50000*
>> *Accept Partial: true*
>> *Is Partial Result: false*
>> *Hit Exception Cache: false*
>> *Storage cache used: false*
>> *Message: null*
>>
>> Maybe i will try LDAP and report result for you.
>>
>> And one more problem that i need your help is: *  "Query performance is
>> not stable"*
>> My query offent take 2-3s. However after a long time i execute that
>> query, it takes about 10s. On log file, i saw that process will load
>> snapshot data again. I try to configure
>> *kylin.table.snapshot.max_mb=1000.*
>> So is this ok?
>>
>> Thanks for your help.
>>
>> 2017-01-10 16:53 GMT+07:00 ShaoFeng Shi <shaofengshi@apache.org>:
>>
>>> Hi Phong,
>>>
>>> Did you enable "LDAP" authentication in Kylin? By default Kylin
>>> configures to use 10MB memory cache only; if LDAP is enabled (it thought be
>>> used in production), will use 2GB memory cache.
>>>
>>> You can try to allocate more memory by:
>>>
>>> cd $KYLIN_HOME
>>> vi ./tomcat/webapps/kylin/WEB-INF/classes/ehcache_test.xml
>>> In there modify "maxBytesLocalHeap="10M"" to a bigger value like 1G
>>>
>>> Then restart Kylin.
>>>
>>> I'm not sure whether it can solve the error, but it worth a try. Please
>>> share the result after apply this. Thanks!
>>>
>>>
>>> 2017-01-10 14:39 GMT+08:00 Phong Pham <phongpham1805@gmail.com>:
>>>
>>>> Hi,
>>>>    I'm still facing with cache problems. After setting:
>>>>
>>>>
>>>>
>>>>
>>>> *kylin.query.cache-threshold-duration=2kylin.query.cache-threshold-scan-count=5000kylin.query.cache-enabled=true*
>>>>
>>>> Result is still:
>>>> *Is Partial Result: false*
>>>> *Hit Exception Cache: false*
>>>> *Storage cache used: false*
>>>>
>>>> i saw some warnings that:
>>>> The configured limit of 1,000 object references was reached while
>>>> attempting to calculate the size of the object graph. Severe performance
>>>> degradation could occur if the sizing operation continues. This can be
>>>> avoided by setting the CacheManger or Cache <sizeOfPolicy> elements
>>>> maxDepthExceededBehavior to "abort" or adding stop points with
>>>> @IgnoreSizeOf annotations. If performance degradation is NOT an issue at
>>>> the configured limit, raise the limit value using the CacheManager or Cache
>>>> <sizeOfPolicy> elements maxDepth attribute. For more information, see
the
>>>> Ehcache configuration documentation.
>>>> 2017-01-10 00:46:48,423 INFO  [Query 25496a78-149a-4752-8c3b-d988a786b0ec-312]
>>>> service.QueryService:284 :
>>>>
>>>> So how can i fix this?
>>>> thanks
>>>>
>>>> 2017-01-10 9:27 GMT+07:00 ShaoFeng Shi <shaofengshi@apache.org>:
>>>>
>>>>> Hi Phong,
>>>>>
>>>>> Kylin cache query results in two scenarios:
>>>>>
>>>>> 1) the query time exceeds 2 seconds (defined by
>>>>> kylin.query.cache-threshold-duration=2000)
>>>>> or 2) the query scan lager than 10 thousands rows in hbase (defined by
>>>>> kylin.query.cache-threshold-scan-count=10240)
>>>>>
>>>>> This means, the very fast and small query will not be hit by cache.
>>>>> You can give smaller values to the above parameters to use cache more
>>>>> frequently.
>>>>>
>>>>> 2017-01-10 9:03 GMT+08:00 Phong Pham <phongpham1805@gmail.com>:
>>>>>
>>>>>> How can i turn on cache for queries?
>>>>>>
>>>>>> I turn on *kylin.query.cache.enabled=true and enabled on System
>>>>>> tab. *
>>>>>> However when i query and submit query again. Result still is:
>>>>>>
>>>>>> Accept Partial: true
>>>>>> Is Partial Result: false
>>>>>> Hit Exception Cache: false
>>>>>> Storage cache used: false
>>>>>>
>>>>>> Kylin Version: 1.6.2
>>>>>> Thanks for your help.
>>>>>>
>>>>>> 2017-01-09 20:36 GMT+07:00 Billy Liu <billyliu@apache.org>:
>>>>>>
>>>>>>> *Hit Exception Cache: false *means your query does not hit not
>>>>>>> cache,  if you query again, the result would be "Hit Exception
Cache: true"
>>>>>>>
>>>>>>> 2017-01-09 17:20 GMT+08:00 Phong Pham <phongpham1805@gmail.com>:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>     i saw this on log file after a query finished:
>>>>>>>>
>>>>>>>> *Accept Partial: true*
>>>>>>>> *Is Partial Result: false*
>>>>>>>> *Hit Exception Cache: false*
>>>>>>>> *Storage cache used: false*
>>>>>>>> *Message: null*
>>>>>>>>
>>>>>>>> On setting, i set:
>>>>>>>> *kylin.query.cache.enabled=true*
>>>>>>>>
>>>>>>>> and enable cache on System tab.
>>>>>>>>
>>>>>>>> So Is my query not cached?
>>>>>>>>
>>>>>>>> And sometime, my kylin server go down with some bad query
or unknow
>>>>>>>> reasons. So how can i kill slow queries automatically?
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>

Mime
View raw message