Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Date: Tue, 23 Jul 2013 06:40:49 +0000 (UTC)
From: "Srikanth Sundarrajan (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12657078.1373466391796.95494.1374561649440@arcas>
In-Reply-To: <JIRA.12657078.1373466391796@arcas>
References: <JIRA.12657078.1373466391796@arcas>
Subject: [jira] [Commented] (FALCON-47) Falcon Replication should support
 configurable delays in feed, parallel, timeout and bulk transfer with
 variable frequency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FALCON-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716158#comment-13716158 ] 

Srikanth Sundarrajan commented on FALCON-47:
--------------------------------------------

Sorry [~mnikhil], accidentally closed this. I marked all JIRAs with fixVersion: 0.3 as closed while preparing for the release of 0.3.

{quote}
In the case of bulk transfers, please also consider to induce a random delay in the range of 0-x, specifiable in seconds/minutes.

I see the delay configured here is going to be static, which is as good as changing the base frequency altogether.

In case of not needing to change the base frequency, I think of randomized delays would be something to look at.
{quote}

Delay feature as implemented in this issue is to initiate the transfer with a phase difference, but doesn't do anything with respect to bulking. I am moving this comment to FALCON-51


> Falcon Replication should support configurable delays in feed, parallel, timeout and bulk transfer with variable frequency
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FALCON-47
>                 URL: https://issues.apache.org/jira/browse/FALCON-47
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.3
>         Environment: oozie 3.x distcp 2.x (custom)
>            Reporter: Shaik Idris Ali
>            Assignee: Shaik Idris Ali
>              Labels: Replication
>             Fix For: 0.3
>
>         Attachments: Falcon-47.patch, Falcon-47-v2.patch
>
>
> Falcon Replication should support configurable delays in feed.
> By default the replication/distcp works without any delay, i.e. as soon as a feed is scheduled, it will start replicating with the current nominal time.
> 1. We need to support usecase of delayed replication, for example, a feed can be produced with a delay of n hours based on the process generating it, however the replication kicks of immediately and oozie goes into waiting state and might timeout.
> 2. We need to support parallel/concurrency for feed replication by capturing it from properties, user may want to run distcp parallelly to backfill data on another cluster.
> 3. timeout can also be set as special property.
> All these can be back-ported from Ivory release.
> 4. Need to support replication with a frequency different from the feed frequency, ex: a feed can have hours(1) as frequency, by default a scheduled feed will distcp every hour, however users should be able to set larger frequency like hours(6) which bulk replicates 6 hours of data in one distcp operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira