falcon-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Sundarrajan <srik...@hotmail.com>
Subject RE: Falcon and S3
Date Mon, 20 Jun 2016 11:11:38 GMT
Yes, Falcon can be used to import / export from s3 and then subsequent processesing can be
orchestrated by falcon.

Ex: feed definition for s3 export/import (This ex: does an export to s3, by switching the
order an import can be realized)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="some-feed-in-s3" description="some-feed-in-s3" xmlns="uri:falcon:feed:0.1">
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(6)"/>
    <clusters>
        <cluster name="my-hdfs-cluster" type="source">
            <validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
            <retention limit="days(30)" action="delete"/>
        </cluster>
        <cluster name="my-s3-mirror" type="target">
            <validity start="2001-01-00T00:00Z" end="2999-12-31T00:00Z"/>
            <retention limit="days(90000)" action="delete"/>
            <locations>
                <location type="data" path="s4://my-s3-bucket-name/feed-in-s3/${YEAR}-${MONTH}-${DAY}-${HOUR}/"/>
            </locations>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/my/hdfs/path/feed-in-s3/${YEAR}/${MONTH}/${DAY}/${HOUR}/"/>
    </locations>
    <ACL owner="my-user" group="group" permission="0x755"/>
    <schema location="/schema/mydata" provider="protobuf"/>
</feed>

To avoid exposing the S3 credentials in the s3-url, you can use the following FS client and
host the credentials in the user's home directory on hdfs.

https://github.com/InMobi/secure-s3-fsclient

Regards
Srikanth Sundarrajan

From: sayatez@gmail.com
Date: Mon, 20 Jun 2016 11:07:22 +0200
Subject: Falcon and S3
To: user@falcon.apache.org

Hi,
I'm a new to Falcon project, but I'm thinking to use it for workflow management tool. I'm
wondering if Falcon integrates with S3 well(i.e. export/import data)?
-- 
Best Regards,
Sayat Satybaldiyev


 		 	   		  
Mime
View raw message