falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-50) Falcon replication having issue to s3
Date Fri, 15 Nov 2013 20:25:21 GMT

    [ https://issues.apache.org/jira/browse/FALCON-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824052#comment-13824052
] 

Venkatesh Seetharam commented on FALCON-50:
-------------------------------------------

I suspect FALCON-85 breaks this since Storage.getUriTemplate() prefixes the storage url from
the cluster definition. It can be done but you'd need another cluster definition with write
being s3. Need to test to confirm this.

> Falcon replication having issue to s3
> -------------------------------------
>
>                 Key: FALCON-50
>                 URL: https://issues.apache.org/jira/browse/FALCON-50
>             Project: Falcon
>          Issue Type: Bug
>          Components: general
>            Reporter: Biswajit Nayak
>              Labels: operability
>             Fix For: 0.4
>
>
> Overview :- Replication of data from one cluster to s3 is not working.
> Detail:-
> Replication feed was submitted and scheduled through falcon. While only the retention
bundle and co-ordinator (in oozie) are kicked up, could not see any sign of replication bundle
or co-ordinator in oozie logs. Dont see any error in logs.
> Below is the feed definition :-
> {noformat}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="process-name" description="process-name" xmlns="uri:falcon:feed:0.1">
>     <frequency>hours(1)</frequency>
>     <late-arrival cut-off="hours(6)" />
>     <clusters>
>         <cluster name="cluster1" type="source">
>             <validity start="2013-07-17T00:00Z" end="2034-12-20T08:00Z"/>
>             <retention limit="days(24)" action="delete"/>
>         </cluster>
>         <cluster name="cluster2" type="target">
>             <validity start="2013-07-17T00:00Z" end="2034-12-20T08:00Z"/>
>             <retention limit="days(90)" action="delete"/>
>             <locations>
>                 <location type="data" path="s3://<bucket-name>/<path-folder>/${YEAR}-${MONTH}-${DAY}-${HOUR}/"/>
>             </locations>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/path/to/copy/${YEAR}/${MONTH}/${DAY}/${HOUR}/"/>
>     </locations>
>     <ACL owner="user" group="group" permission="0x755"/>
>     <schema location="/schema/s1" provider="protobuf"/>
>     <properties>
>         <property name="queueName" value="queueName"/>
>         <property name="jobPriority" value="NORMAL"/>
>     </properties>
> </feed>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message