flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Zhekai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLUME-3341) Taildir source may cause file handle leak and data duplication
Date Sun, 18 Aug 2019 14:25:00 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ming Zhekai updated FLUME-3341:
    Attachment:     (was: ReliableTaildirEventReader.java)

> Taildir source may cause file handle leak and data duplication
> --------------------------------------------------------------
>                 Key: FLUME-3341
>                 URL: https://issues.apache.org/jira/browse/FLUME-3341
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: Ming Zhekai
>            Priority: Major
>             Fix For: 1.8.0
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> As is described in Flume-3342, renaming a file may cause data duplication. But moreover,
if the original file was opened by flume before and not closed yet, flume reopens it while
not freeing its original file handles. And then I delete the new file, but after the idle
time, flume just closes the new file handle and forgets to close its original file handles,
leading to a handle leak. It can be found by "lsof | grep "deleted".
> This bug is triggered by using  Log4j to roll log files. To avoid losing data when
rolling, I use regex to include both current log file and old log files in the taildir path.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message