falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-47) Falcon Replication should support configurable delays in feed, parallel, timeout and bulk transfer with variable frequency
Date Mon, 15 Jul 2013 02:22:49 GMT

    [ https://issues.apache.org/jira/browse/FALCON-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708206#comment-13708206

Srikanth Sundarrajan commented on FALCON-47:

Is there a reason, why will override user-provided value with 30 minutes if it is less than
30 for timeout ?

+            Map<String, String> props = getEntityProperties();
+            String timeout = props.get(TIMEOUT);
+            if (timeout!=null) {
+                try{
+                    timeoutInMillis= ExpressionHelper.get().
+                            evaluate(timeout, Long.class);
+                } catch (Exception ignore) {
+                    LOG.error("Unable to evaluate timeout:", ignore);
+                }
+            }
             if (timeoutInMillis < THIRTY_MINUTES) {
                 timeoutInMillis = THIRTY_MINUTES;

It might be better to also add entries to .gitignore for eclipse files.

Also please update documentation relating to this feature and expected behavior in https://svn.apache.org/repos/asf/incubator/falcon.
> Falcon Replication should support configurable delays in feed, parallel, timeout and
bulk transfer with variable frequency
> --------------------------------------------------------------------------------------------------------------------------
>                 Key: FALCON-47
>                 URL: https://issues.apache.org/jira/browse/FALCON-47
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.3
>         Environment: oozie 3.x distcp 2.x (custom)
>            Reporter: Shaik Idris Ali
>              Labels: Replication
>         Attachments: Falcon-47.patch
> Falcon Replication should support configurable delays in feed.
> By default the replication/distcp works without any delay, i.e. as soon as a feed is
scheduled, it will start replicating with the current nominal time.
> 1. We need to support usecase of delayed replication, for example, a feed can be produced
with a delay of n hours based on the process generating it, however the replication kicks
of immediately and oozie goes into waiting state and might timeout.
> 2. We need to support parallel/concurrency for feed replication by capturing it from
properties, user may want to run distcp parallelly to backfill data on another cluster.
> 3. timeout can also be set as special property.
> All these can be back-ported from Ivory release.
> 4. Need to support replication with a frequency different from the feed frequency, ex:
a feed can have hours(1) as frequency, by default a scheduled feed will distcp every hour,
however users should be able to set larger frequency like hours(6) which bulk replicates 6
hours of data in one distcp operation.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message