nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Burke <kyle.bu...@ignitionone.com>
Subject Re: ListS3 processor?
Date Sat, 30 Jan 2016 19:52:54 GMT
Joe/Joe,
  Thanks for the response. It makes sense to use SNS and SQS to respond to S3 file changes.
I’m going see if my company will give me access to those Amazon services. I found an article
that explains how to setup on this functionality in the Amazon console. Once that’s setup
it seems pretty straight forward to use GetSQS/DeleteSQS. I suspect many will want this functionality
but I’m not sure what’s the best method (i.e. Template or user doc) that explains how
to solve this in nifi.  I’ll be happy to submit something if you let me know the right method.

http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Respectfully,

Kyle Burke | Data Science Engineer
IgnitionOne - Marketing Technology. Simplified.
Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309


From: Joe Witt
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
Date: Saturday, January 30, 2016 at 2:06 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
Subject: Re: ListS3 processor?


Kyle

Let us know if that doesn't get you what you need.  We have a decent set of templates but
I didn't see one that demonstrates interaction with amazon services.

Thanks
Joe

On Jan 30, 2016 12:56 PM, "Joey Frazee" <joey.frazee@icloud.com<mailto:joey.frazee@icloud.com>>
wrote:
Kyle,

I think you can do what you want right now without ListS3 by using S3 event notifications.
You can configure an event notification to publish to SQS and then use GetSQS to retrieve
the events and FetchS3Object to get the JSON file and the rest of the flow could be written
as you have in mind.

Depending on your scale, this might be preferable because it's slow/expensive to do listings
on S3 prefixes that have a lot of file matches.


-joey

On Jan 30, 2016, at 11:40 AM, Joe Skora <jskora@gmail.com<mailto:jskora@gmail.com>>
wrote:

Kyle,

Processors exist to Put, Fetch, and Delete S3Objects, but ListS3 is in the backlog on ticket
NIFI-840<https://issues.apache.org/jira/browse/NIFI-840> at the moment.  It should fit
the List/Fetch metaphor like the List/Fetch processors pairs for xFile, xHDFS, xSFTP, etc.

Regards,
Joe Skora

On Sat, Jan 30, 2016 at 10:14 AM, Kyle Burke <kyle.burke@ignitionone.com<mailto:kyle.burke@ignitionone.com>>
wrote:
All,
  I'm trying to get Nifi set up to a move data around S3. My first attempt is to just monitor
a S3 folder where json files are placed and then copy the file, convert it to Avro, and the
drop it in a different S3 folder. The documentation is pretty slim for working with S3. I
can't seem to get it working and was wondering if anyone had any S3 examples for monitoring
an S3 folder (i.e.. something like a ListS3 processer similar to what is available on a local
file system?)

Respectfully,

Kyle Burke | Data Science Engineer
IgnitionOne - Marketing Technology. Simplified.
Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
Direct: 404.961.3918<tel:404.961.3918>


Mime
View raw message