kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vsevolod Ostapenko (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KYLIN-3122) Partition elimination algorithm seems to be inefficient and have serious issues with handling date/time ranges, can lead to very slow queries and OOM/Java heap dump conditions
Date Mon, 29 Jan 2018 17:32:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343677#comment-16343677
] 

Vsevolod Ostapenko edited comment on KYLIN-3122 at 1/29/18 5:31 PM:
--------------------------------------------------------------------

[~Shaofengshi], we tried using single partitioning column with date and time fused together.
 The result is still not satisfactory, as cube segments are not properly eliminated even in
this case.

In our test we had a table with 12 hourly partitions defined for hours 00 to 11. A test query
with condition _*where a.time_key >= '201711200100' and a.time_key < '201711200400'*_ 
is only filtering out the very fist segment for the hour 00, and then progresses with scanning
all the remaining 11 segments, instead of the expected 3 segments (for hours 01, 02 and 03).
 It looks very much like a bug, where as soon as the lower bound condition is satisfied, upper
bound condition is no longer checked.

I'm attaching a log excerpt to illustrate the above mentioned behavior.

[^partition_elimination_bug_single_column_test.log]


was (Author: seva_ostapenko):
[~Shaofengshi], we tried using single partitioning column with date and time fused together.
The result is still not satisfactory, as cube segments are not properly eliminated even in
this case.

In our test we had a table with 12 hourly partitions defined for hours 00 to 11, a query with
condition _*where a.time_key >= '201711200100' and a.time_key < '201711200400'*_ 
is only filtering out the very fist segment for hour 00, then progresses with scanning all
the remaining 11 segments, instead of expected 3 segments (hours 01, 02 and 03).
It looks very much like a bug, where as soon as the lower bound condition is satisfied, upper
bound condition is no longer checked.

I'm attaching a log excerpt to illustrate the above mentioned behavior.

[^partition_elimination_bug_single_column_test.log]

> Partition elimination algorithm seems to be inefficient and have serious issues with
handling date/time ranges, can lead to very slow queries and OOM/Java heap dump conditions
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-3122
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3122
>             Project: Kylin
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: v2.2.0
>         Environment: HDP 2.5.6, Kylin 2.2.0
>            Reporter: Vsevolod Ostapenko
>            Assignee: hongbin ma
>            Priority: Critical
>         Attachments: partition_elimination_bug_single_column_test.log
>
>
> Current algorithm of cube segment elimination seems to be rather inefficient.
>  We are using a model where cubes are partitioned by date and time:
>  "partition_desc":
> { "partition_date_column": "A_VL_HOURLY_V.THEDATE", "partition_time_column": "A_VL_HOURLY_V.THEHOUR",
"partition_date_start": 0, "partition_date_format": "yyyyMMdd", "partition_time_format": "HH",
"partition_type": "APPEND", "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
}
> ,
> Cubes contain partitions for multiple days and 24 hours for each day. Each cube segment
corresponds to just one hour.
> When a query is issued where both date and hour are specified using equality condition
(e.g. thedate = '20171011' and thehour = '10') Kylin sequentially integrates over all the
segment cubes (hundreds of them) only to skip all except for the one that needs to be scanned
(which can be observed by looking in the logs).
>  The expectation is that Kylin would use existing info on the partitioning columns (date
and time) and known hierarchical relations between date and time to locate required partition
much more efficiently that linear scan through all the cube partitions.
> Now, if filtering condition is on the range of hours, behavior of the partition pruning
and scanning becomes not very logical, which suggests bugs in the logic.
> If filtering condition is on specific date and closed-open range of hours (e.g. thedate
= '20171011' and thehour >= '10' and thehour < '11'), in addition to sequentially scanning
all the cube partitions (as described above), Kylin will scan HBase tables for all the hours
from the specified starting hour and till the last hour of the day (e.g. from hour 10 to 24,
instead of just hour 10).
>  As the result query will run much longer that necessary, and might run out of memory,
causing JVM heap dump and Kylin server crash.
> If filtering condition is on specific date by hour interval is specified as open-closed
(e.g. thedate = '20171011' and thehour > '09' and thehour <= '10'), Kylin will scan
all HBase tables for all the later dates and hours (e.g. from hour 10 and till the most recent
hour on the most recent day, which can be hundreds of tables and thousands of regions).
>  As the result query execution will dramatically increase and in most cases Kylin server
will be terminated with OOM error and JVM heap dump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message