flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hunshenshi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLUME-3101) taildir source may endless loop when tail a file
Date Thu, 01 Jun 2017 10:14:04 GMT
hunshenshi created FLUME-3101:

             Summary: taildir source may endless loop when tail a file
                 Key: FLUME-3101
                 URL: https://issues.apache.org/jira/browse/FLUME-3101
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: 1.7.0
            Reporter: hunshenshi

If there are many files in the path that need to be tail, and there is a file written by *high
frequency* (for example, there are file a, file b and file c in the path, file a is written
at high frequency), *taildir can read the batchSize size event from file a everytime*, then
taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess
will into an endless loop.

private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
    throws IOException, InterruptedException {
  while (true) {
    // if events.size >= batchSize will not break while, 
    // then into endless loop to only read tf
    if (events.size() < batchSize) {

This message was sent by Atlassian JIRA

View raw message