nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NIFI-2631) ListS3 improvements: "Use versions" and "Commit mode"
Date Fri, 02 Sep 2016 16:26:20 GMT


ASF GitHub Bot commented on NIFI-2631:

Github user jvwing commented on the issue:
    @gresockj No, I have not noticed that, and I've been viewing and clearing the state along
with my testing.  Are there javascript errors in your browser? Are you familiar with browser
debugging tools?  When you open the state dialog, there should be an XHR call to the server
to get this state information, something like:
    And this returns JSON with the state in it.  Can you check if that query is being run,
and if it is failing or returning nothing?

> ListS3 improvements: "Use versions" and "Commit mode"
> -----------------------------------------------------
>                 Key: NIFI-2631
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 0.7.0
>            Reporter: Joseph Gresock
>            Assignee: Joseph Gresock
>            Priority: Minor
>             Fix For: 1.1.0, 0.8.0
> Our team needs to be able to list individual versions in S3.  We also ran into a use
case where a bucket with many objects (over 1 million in our case) seemed to cause ListS3
to run forever.  The S3 list command finished in a few minutes, but we believe it was taking
a very long time for NiFi to commit all the flow files at once.
> To handle this use case, we added a Commit Mode property to ListS3 that allows you specify
that you want to commit "Per page" vs. "Once".  This has proven to correctly emit the flow
files as the S3 paging progresses.
> We also implemented support for S3 List Versions, which includes the "s3.version" and
"s3.isLatest" attributes if applicable.  The "s3.version" attribute can in turn be used in
the FetchS3 processor.

This message was sent by Atlassian JIRA

View raw message