kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3140) Auto merge jobs should not block user build jobs
Date Fri, 02 Feb 2018 05:42:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349817#comment-16349817
] 

Shaofeng SHI commented on KYLIN-3140:
-------------------------------------

Hi Gang, I understand your point.

The end user may not care how many merge job running, but the platform administrator cares.
The "max-building-segments" was introduced to contraint the max concurrent jobs a Kylin server
can run. If exceed this threshold, the server may encounter performance issue and report errors
like OOM. That's why it doesn't differentiate the merge/refresh and build.

Usually if a job is failed, the administrator need fix the problem immediately and then decide
to resume or discard it. If the root cause isn't identified, resume or discard doesn't help.
Even for merge job, the administrator need take care, we don't see a strong reason to ignore
that.

 

> Auto merge jobs should not block user build jobs
> ------------------------------------------------
>
>                 Key: KYLIN-3140
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3140
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>            Reporter: Wang, Gang
>            Assignee: Shaofeng SHI
>            Priority: Major
>
> Although in the latest version, Kylin support concurrent jobs. If the concurrency is
set to 1, there is some possibility that cube build job will have dead lock. Say, when there
is some issue which causes merge job failed, even when you discard the job, another job will
be launched and failed again due to auto merge policy. And this failed merge job blocks user
to build incremental segment.
> Even if the concurrency is set to larger than 1, the auto merge jobs occupy some concurrency
quota. 
> While, from user perspective, they don't care much about the auto merge jobs, and the
auto merge jobs should not block the building/refresh jobs they submit manually.
> A better way may be separating the auto merge jobs from the job queue, parameter max-building-segments
only limit jobs submitted by users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message