nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-1170) TailFile "File to Tail" property should support Wildcards
Date Mon, 12 Sep 2016 19:26:20 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485040#comment-15485040
] 

ASF GitHub Bot commented on NIFI-1170:
--------------------------------------

Github user pvillard31 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/980#discussion_r78437746
  
    --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
---
    @@ -245,58 +490,82 @@ private void recoverState(final ProcessContext context, final Map<String,
String
                             + "this indicates that the file has rotated. Will begin tailing
current file from beginning.", new Object[]{existingTailFile.length(), position});
                 }
     
    -            state = new TailFileState(currentFilename, tailFile, reader, position, timestamp,
checksum, ByteBuffer.allocate(65536));
    +            states.get(filePath).setState(new TailFileState(filePath, tailFile, reader,
position, timestamp, length, checksum, ByteBuffer.allocate(65536)));
             } else {
    -            resetState(currentFilename);
    +            resetState(filePath);
             }
     
    -        getLogger().debug("Recovered state {}", new Object[]{state});
    +        getLogger().debug("Recovered state {}", new Object[]{states.get(filePath).getState()});
         }
     
    -    private void resetState(final String currentFilename) {
    -        expectedRecoveryChecksum = null;
    -        state = new TailFileState(currentFilename, null, null, 0L, 0L, null, ByteBuffer.allocate(65536));
    +    private void resetState(final String filePath) {
    +        states.get(filePath).setExpectedRecoveryChecksum(null);
    +        states.get(filePath).setState(new TailFileState(filePath, null, null, 0L, 0L,
0L, null, ByteBuffer.allocate(65536)));
         }
     
         @OnStopped
         public void cleanup() {
    -        final TailFileState state = this.state;
    -        if (state == null) {
    +        for (TailFileObject tfo : states.values()) {
    +            cleanReader(tfo);
    +            final TailFileState state = tfo.getState();
    +            tfo.setState(new TailFileState(state.getFilename(), state.getFile(), null,
state.getPosition(), state.getTimestamp(), state.getLength(), state.getChecksum(), state.getBuffer()));
    +        }
    +    }
    +
    +    private void cleanReader(TailFileObject tfo) {
    +        if (tfo.getState() == null) {
                 return;
             }
     
    -        final FileChannel reader = state.getReader();
    +        final FileChannel reader = tfo.getState().getReader();
             if (reader == null) {
                 return;
             }
     
             try {
                 reader.close();
    +            getLogger().debug("Closed FileChannel {}", new Object[]{reader});
             } catch (final IOException ioe) {
                 getLogger().warn("Failed to close file handle during cleanup");
             }
    -
    -        getLogger().debug("Closed FileChannel {}", new Object[]{reader});
    -
    -        this.state = new TailFileState(state.getFilename(), state.getFile(), null, state.getPosition(),
state.getTimestamp(), state.getChecksum(), state.getBuffer());
         }
     
         @Override
         public void onTrigger(final ProcessContext context, final ProcessSession session)
throws ProcessException {
    +        if(isMultiChanging.get()) {
    +            long timeSinceLastLookup = new Date().getTime() - lastLookup.get();
    +            if(timeSinceLastLookup > context.getProperty(LOOKUP_FREQUENCY).asTimePeriod(TimeUnit.MILLISECONDS))
{
    +                try {
    +                    recoverState(context);
    +                } catch (IOException e) {
    +                    getLogger().warn("Exception raised while looking up for new files",
e);
    --- End diff --
    
    Good catch.


> TailFile "File to Tail" property should support Wildcards
> ---------------------------------------------------------
>
>                 Key: NIFI-1170
>                 URL: https://issues.apache.org/jira/browse/NIFI-1170
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 0.4.0
>            Reporter: Andre
>
> Because of challenges around log rotation of high volume syslog and app producers, it
is customary to logging platform developers to promote file variables based file names such
as DynaFiles (rsyslog), Macros(syslog-ng)as alternatives to getting SIGHUPs being sent to
the syslog daemon upon every file rotation.
> (To certain extent, used even NiFi's has similar patterns, like for example, when one
uses Expression Language to set PutHDFS destination file).
> The current TailFile strategy suggests rotation patterns like:
> {code}
> log_folder/app.log
> log_folder/app.log.1
> log_folder/app.log.2
> log_folder/app.log.3
> {code}
> It is possible to fool the system to accept wildcards by simply using a strategy like:
> {code}
> log_folder/test1
> log_folder/server1
> log_folder/server2
> log_folder/server3
> {code}
> And configure *Rolling Filename Pattern* to * but it feels like a hack, rather than catering
for an ever increasingly prevalent use case (DynaFile/macros/etc).
> It would be great if instead, TailFile had the ability to use wildcards on File to Tail
property



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message