falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-47) Falcon Replication should support configurable delays in feed, parallel, timeout and bulk transfer with variable frequency
Date Mon, 15 Jul 2013 02:24:48 GMT

    [ https://issues.apache.org/jira/browse/FALCON-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708206#comment-13708206
] 

Srikanth Sundarrajan edited comment on FALCON-47 at 7/15/13 2:23 AM:
---------------------------------------------------------------------

Is there a reason, why will override user-provided value with 30 minutes if it is less than
30 for timeout ?

{code}
+            Map<String, String> props = getEntityProperties();
+            String timeout = props.get(TIMEOUT);
+            if (timeout!=null) {
+                try{
+                    timeoutInMillis= ExpressionHelper.get().
+                            evaluate(timeout, Long.class);
+                } catch (Exception ignore) {
+                    LOG.error("Unable to evaluate timeout:", ignore);
+                }
+            }
             if (timeoutInMillis < THIRTY_MINUTES) {
                 timeoutInMillis = THIRTY_MINUTES;
             }
{code}

It might be better to also add entries to .gitignore for eclipse files.

Also please update documentation relating to this feature and expected behavior in https://svn.apache.org/repos/asf/incubator/falcon.
                
      was (Author: sriksun):
    Is there a reason, why will override user-provided value with 30 minutes if it is less
than 30 for timeout ?

+            Map<String, String> props = getEntityProperties();
+            String timeout = props.get(TIMEOUT);
+            if (timeout!=null) {
+                try{
+                    timeoutInMillis= ExpressionHelper.get().
+                            evaluate(timeout, Long.class);
+                } catch (Exception ignore) {
+                    LOG.error("Unable to evaluate timeout:", ignore);
+                }
+            }
             if (timeoutInMillis < THIRTY_MINUTES) {
                 timeoutInMillis = THIRTY_MINUTES;
             }

It might be better to also add entries to .gitignore for eclipse files.

Also please update documentation relating to this feature and expected behavior in https://svn.apache.org/repos/asf/incubator/falcon.
                  
> Falcon Replication should support configurable delays in feed, parallel, timeout and
bulk transfer with variable frequency
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FALCON-47
>                 URL: https://issues.apache.org/jira/browse/FALCON-47
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.3
>         Environment: oozie 3.x distcp 2.x (custom)
>            Reporter: Shaik Idris Ali
>              Labels: Replication
>         Attachments: Falcon-47.patch
>
>
> Falcon Replication should support configurable delays in feed.
> By default the replication/distcp works without any delay, i.e. as soon as a feed is
scheduled, it will start replicating with the current nominal time.
> 1. We need to support usecase of delayed replication, for example, a feed can be produced
with a delay of n hours based on the process generating it, however the replication kicks
of immediately and oozie goes into waiting state and might timeout.
> 2. We need to support parallel/concurrency for feed replication by capturing it from
properties, user may want to run distcp parallelly to backfill data on another cluster.
> 3. timeout can also be set as special property.
> All these can be back-ported from Ivory release.
> 4. Need to support replication with a frequency different from the feed frequency, ex:
a feed can have hours(1) as frequency, by default a scheduled feed will distcp every hour,
however users should be able to set larger frequency like hours(6) which bulk replicates 6
hours of data in one distcp operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message