nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NIFI-1170) TailFile "File to Tail" property should support Wildcards
Date Mon, 12 Sep 2016 19:29:22 GMT


ASF GitHub Bot commented on NIFI-1170:

Github user pvillard31 commented on a diff in the pull request:
    --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/
    @@ -117,31 +173,78 @@
                 .allowableValues(LOCATION_LOCAL, LOCATION_REMOTE)
         static final PropertyDescriptor START_POSITION = new PropertyDescriptor.Builder()
                 .name("Initial Start Position")
    -            .description("When the Processor first begins to tail data, this property
specifies where the Processor should begin reading data. Once data has been ingested from
the file, "
    +            .description("When the Processor first begins to tail data, this property
specifies where the Processor should begin reading data. Once data has been ingested from
a file, "
                         + "the Processor will continue from the last point from which it
has received data.")
    +    static final PropertyDescriptor RECURSIVE = new PropertyDescriptor.Builder()
    +            .name("tailfile-recursive-lookup")
    +            .displayName("Recursive lookup")
    +            .description("When using Multiple files mode, this property defines if files
must be listed recursively or not"
    +                    + " in the base directory.")
    +            .allowableValues("true", "false")
    +            .defaultValue("true")
    +            .required(true)
    +            .build();
    +    static final PropertyDescriptor ROLLING_STRATEGY = new PropertyDescriptor.Builder()
    +            .name("tailfile-rolling-strategy")
    +            .displayName("Rolling Strategy")
    +            .description("Specifies if the files to tail have a fixed name or not.")
    +            .required(true)
    +            .allowableValues(FIXED_NAME, CHANGING_NAME)
    +            .defaultValue(FIXED_NAME.getValue())
    +            .build();
    +    static final PropertyDescriptor LOOKUP_FREQUENCY = new PropertyDescriptor.Builder()
    +            .name("tailfile-lookup-frequency")
    +            .displayName("Lookup frequency")
    +            .description("Only used in Multiple files mode and Changing name rolling
strategy, it specifies the minimum "
    +                    + "duration the processor will wait before listing again the files
to tail.")
    +            .required(false)
    +            .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
    +            .defaultValue("10 minutes")
    --- End diff --
    You are definitely right, it really depends of the use case and I am in no position to
assume that those values are a good fit in most situations. I removed default values and I'll
assume that additional details in the documentation will give a good sense of what is the
role of those values.

> TailFile "File to Tail" property should support Wildcards
> ---------------------------------------------------------
>                 Key: NIFI-1170
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 0.4.0
>            Reporter: Andre
> Because of challenges around log rotation of high volume syslog and app producers, it
is customary to logging platform developers to promote file variables based file names such
as DynaFiles (rsyslog), Macros(syslog-ng)as alternatives to getting SIGHUPs being sent to
the syslog daemon upon every file rotation.
> (To certain extent, used even NiFi's has similar patterns, like for example, when one
uses Expression Language to set PutHDFS destination file).
> The current TailFile strategy suggests rotation patterns like:
> {code}
> log_folder/app.log
> log_folder/app.log.1
> log_folder/app.log.2
> log_folder/app.log.3
> {code}
> It is possible to fool the system to accept wildcards by simply using a strategy like:
> {code}
> log_folder/test1
> log_folder/server1
> log_folder/server2
> log_folder/server3
> {code}
> And configure *Rolling Filename Pattern* to * but it feels like a hack, rather than catering
for an ever increasingly prevalent use case (DynaFile/macros/etc).
> It would be great if instead, TailFile had the ability to use wildcards on File to Tail

This message was sent by Atlassian JIRA

View raw message