helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei Xia (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HELIX-617) Job IdealState is generated even the job is not running and not removed when it is completed.
Date Wed, 09 Dec 2015 22:00:12 GMT
Lei Xia created HELIX-617:
-----------------------------

             Summary: Job IdealState is generated even the job is not running and not removed
when it is completed.
                 Key: HELIX-617
                 URL: https://issues.apache.org/jira/browse/HELIX-617
             Project: Apache Helix
          Issue Type: Task
            Reporter: Lei Xia
            Assignee: Lei Xia


-- Problem:
In current task framework implementation, one IdealState and one ExternalView znode are generated
for each job once the job is queued (not when it is started). These znodes will stay in the
zookeeper until the specified data expiry time (usually one day in Espresso backup setting).
This is even worse for recurrent job queue.  For recurrent job queue, one idealstate/externalview
is created for original job template, and one idealstate/externalview will be created for
each scheduled run of the job. 

This usually generates a significant amounts of znodes in IdealState.  For example, to schedule
a queue with 100 jobs, 100 "template" idealstates are created and will stay there for ever.
While a new set of 100 "scheduling" idealstates are created for every time all these jobs
are scheduled to run, which will stay until the expiry time setup in the job configuration
has passed (usually a relative long time).

-- Proposed Changes:
The IdealState of a job will be generated only when it is scheduled to run, and will be removed
immediately once the job is completed.  In practice, there will be only one job running at
the same time for each queue, thus only one IdealState will exist for each queue at any given
time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message