nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pvillard31 <...@git.apache.org>
Subject [GitHub] nifi pull request #980: NIFI-1170 - Improved TailFile processor to support m...
Date Thu, 15 Sep 2016 12:44:53 GMT
Github user pvillard31 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/980#discussion_r78954211
  
    --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
---
    @@ -117,31 +173,78 @@
                 .allowableValues(LOCATION_LOCAL, LOCATION_REMOTE)
                 .defaultValue(LOCATION_LOCAL.getValue())
                 .build();
    +
         static final PropertyDescriptor START_POSITION = new PropertyDescriptor.Builder()
                 .name("Initial Start Position")
    -            .description("When the Processor first begins to tail data, this property
specifies where the Processor should begin reading data. Once data has been ingested from
the file, "
    +            .description("When the Processor first begins to tail data, this property
specifies where the Processor should begin reading data. Once data has been ingested from
a file, "
                         + "the Processor will continue from the last point from which it
has received data.")
                 .allowableValues(START_BEGINNING_OF_TIME, START_CURRENT_FILE, START_CURRENT_TIME)
                 .defaultValue(START_CURRENT_FILE.getValue())
                 .required(true)
                 .build();
     
    +    static final PropertyDescriptor RECURSIVE = new PropertyDescriptor.Builder()
    +            .name("tailfile-recursive-lookup")
    +            .displayName("Recursive lookup")
    +            .description("When using Multiple files mode, this property defines if files
must be listed recursively or not"
    +                    + " in the base directory.")
    +            .allowableValues("true", "false")
    +            .defaultValue("true")
    +            .required(true)
    +            .build();
    +
    +    static final PropertyDescriptor ROLLING_STRATEGY = new PropertyDescriptor.Builder()
    +            .name("tailfile-rolling-strategy")
    +            .displayName("Rolling Strategy")
    +            .description("Specifies if the files to tail have a fixed name or not.")
    +            .required(true)
    +            .allowableValues(FIXED_NAME, CHANGING_NAME)
    +            .defaultValue(FIXED_NAME.getValue())
    +            .build();
    +
    +    static final PropertyDescriptor LOOKUP_FREQUENCY = new PropertyDescriptor.Builder()
    +            .name("tailfile-lookup-frequency")
    +            .displayName("Lookup frequency")
    +            .description("Only used in Multiple files mode and Changing name rolling
strategy, it specifies the minimum "
    +                    + "duration the processor will wait before listing again the files
to tail.")
    +            .required(false)
    +            .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
    +            .defaultValue("10 minutes")
    --- End diff --
    
    @olegz @joewitt 
    Just updated the PR with a default of 10 minutes for refresh frequency, and 24 hours for
maximum age as a daily rollover pattern seems to be indeed the most common case. I updated
the property descriptions and additional details page.
    I also changed a little bit the approach on the regex matching while listing files to
tail. Previously I was only matching the regular expression against file name, now I am trying
to match the regular expression against the path. It gives more flexibility to the users.
I added an example in the additional details.
    And I squashed my commits.
    Let me know your thoughts.
    
    @trixpan, I understand your point but it sounds like an edge case, no? It would require
to tail a very large number of files to get in such a situation. I'd be inclined to see this
as an additional improvement in a separate PR if you agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message