flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hunshenshi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLUME-3101) taildir source may endless loop when tail a file
Date Thu, 01 Jun 2017 10:14:04 GMT
hunshenshi created FLUME-3101:
---------------------------------

             Summary: taildir source may endless loop when tail a file
                 Key: FLUME-3101
                 URL: https://issues.apache.org/jira/browse/FLUME-3101
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: 1.7.0
            Reporter: hunshenshi


If there are many files in the path that need to be tail, and there is a file written by *high
frequency* (for example, there are file a, file b and file c in the path, file a is written
at high frequency), *taildir can read the batchSize size event from file a everytime*, then
taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess
will into an endless loop.

code:
{code:title=Bar.java|borderStyle=solid}
private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
    throws IOException, InterruptedException {
  while (true) {
    // if events.size >= batchSize will not break while, 
    // then into endless loop to only read tf
    if (events.size() < batchSize) {
      break;
    }
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message