flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "belvey (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLUME-3325) TaildirSource causing oom when tailing a lot of files, where some file is writing frequently but the others are idle.
Date Tue, 26 Mar 2019 15:15:00 GMT
belvey created FLUME-3325:
-----------------------------

             Summary: TaildirSource causing oom when tailing a lot of files, where some file
is writing frequently but the others are idle.
                 Key: FLUME-3325
                 URL: https://issues.apache.org/jira/browse/FLUME-3325
             Project: Flume
          Issue Type: Bug
    Affects Versions: 1.8.0, 1.7.0
            Reporter: belvey


I am using taildir source to read file to kafka, but recently i am facing flume agent oom
error.

With a deap dive to OOM dump file, i found 'List<Long> idleInodes ' of TaildirSource
class  occupied most of the memory. so i looked into the source code of TaildirSource, 
and found that when tailing a lot of files where some file is writing frequently but the
others are idle  can cause the idleInodes List  increasing infinitely.

Relative code is shown below:

 
{code:java}
public Status process() {
Status status = Status.BACKOFF;
try {
existingInodes.clear();
existingInodes.addAll(reader.updateTailFiles());
for (long inode : existingInodes) {
TailFile tf = reader.getTailFiles().get(inode);
if (tf.needTail()) {
boolean hasMoreLines = tailFileProcess(tf, true);
if (hasMoreLines) {
status = Status.READY;
}
}
}
closeTailFiles();
} catch (Throwable t) {
logger.error("Unable to tail files", t);
sourceCounter.incrementEventReadFail();
status = Status.BACKOFF;
}
return status;
}
{code}
when one file is frequently writing, this thread is blocked at this line , and will not 
execute closeTailFiles() method to clear the idleInodes List.
{code:java}
boolean hasMoreLines = tailFileProcess(tf, true);{code}
 

while the other files are idle, and there is anthor thread checking for idle file every 5
sec by default and if one file is always idle ,it may be add to list multi times, so this
list can grow infinitely.

I will add a pull request to it later.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message