kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-3335) Add project & cube related info to the job id for better filtering
Date Mon, 25 Jun 2018 03:29:00 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shaofeng SHI updated KYLIN-3335:
--------------------------------
    Fix Version/s: v2.5.0

> Add project & cube related info to the job id for better filtering
> ------------------------------------------------------------------
>
>                 Key: KYLIN-3335
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3335
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Zhong Yanghong
>            Priority: Major
>             Fix For: v2.5.0
>
>
> Currently it's painful to search cube or project related jobs, since those infos are
hidden in values. Especially, when users want to list job in a period under one project, by
current design, all of the job output info have to be read into memory. If this kind of operation
is done very often within a short period, it's easy to get OOM.
> If the job id is prefixed with project and cube names, then we can push down prefix filters,
which is efficient and safe.
> This kind of change will cause backward compatibility issue. There're two ways to deal
with this:
> * Set a milestone with time tag, if a search relates to data earlier than this time,
then just need to do one scan with prefix filter. Otherwise, two scans are needed. One with
prefix filter and the other use current strategy. As time goes on, old job infos will be deleted.
Once there's no data older than the time, only one scan is needed.
> * Do migration for the old data once.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message