falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Ramesh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1174) Ability to disable oozie dryrun while scheduling or updating the falcon entity
Date Tue, 12 May 2015 23:02:00 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14540959#comment-14540959

Sowmya Ramesh commented on FALCON-1174:

[~shwethags], [~pallavi.rao]: I agree that dry run feature helps to finds the issues early
as it is sync. Default value is always not to skip the dry run.

One of the HDP customer had issue with using latest(0) with hive partition and root cause
was that Oozie dry run had some bug and was failing. Workaround was to provide the knob to
customer to enable or disable the dry run.

I can think of couple of reasons why this can be helpful
* If I am not wrong in Oozie dry run and running a job are two separate features. I think
its good to provide the same flexibility even in Falcon
* If user wants to skip the dry run to enhance performance - I know its won't be a huge gain
but it's nice to provide this flag to enable or disable the dry run. It's similar to skip
checksum option provided in DistCP. 

Since by default dry run is enabled I don't think it will introduce correctness issue. We
can include this in documentation too if user decides to skip the checksum. This is just "nice
to have".

> Ability to disable oozie dryrun while scheduling or updating the falcon entity
> ------------------------------------------------------------------------------
>                 Key: FALCON-1174
>                 URL: https://issues.apache.org/jira/browse/FALCON-1174
>             Project: Falcon
>          Issue Type: Improvement
>          Components: oozie
>    Affects Versions: 0.6
>            Reporter: Sowmya Ramesh
>            Assignee: Sowmya Ramesh
>             Fix For: 0.7
>         Attachments: FALCON-1174.v0.patch
> Today, before scheduling or updating an entity Oozie dry run is done to validate it.
This validation helps as Oozie dry run is s a sync operation and hence validation can be done
before scheduling the job.
> There may be a requirement where user would want to turn off this feature. Improvement
is to provide a config knob to skip the dry run.

This message was sent by Atlassian JIRA

View raw message