flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3223) Flume HDFS Sink should retry close prior to performing a recoverLease
Date Mon, 15 Oct 2018 09:20:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16649953#comment-16649953

ASF subversion and git services commented on FLUME-3223:

Commit 327a43dbefab57dc5d2da3ca9b7bb36ae4f743b7 in flume's branch refs/heads/trunk from [~emajor]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=327a43d ]

FLUME-3223 Flume HDFS Sink should retry close prior recover lease

This is based on @mcsanady 's original pull request #202
I took the test changes from him but reworked the new feature implementation
since it failed some unit tests.
Previously when a close failed we immediately did a recover lease.
This PR introduces a background retry mechanism. It uses the already
existing "hdfs.closeTries" parameter. Unfortunately it has infinite retries
by default, that seems a bit too long for me.

I also did a minimal code clean up. The most important is that
HDFSWriter writer in BucketWriter became final. This is essential for later use
in inner classes. Only some testing solutions made it not final.
I reworked those to use the constructor.

This closes #229
This closes #202

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)

> Flume HDFS Sink should retry close prior to performing a recoverLease
> ---------------------------------------------------------------------
>                 Key: FLUME-3223
>                 URL: https://issues.apache.org/jira/browse/FLUME-3223
>             Project: Flume
>          Issue Type: Improvement
>            Reporter: Miklos Csanady
>            Assignee: Endre Major
>            Priority: Major
> Flume HDFS Sink will currently only attempt to close a file once, and if that fails,
it will move to recoverLease. This appears to be causing some issues.
> The recommendation is to modify the close behavior to retry the close a configurable
number of times with a configurable interval in between before attempting the recoverLease.
> There currently exists a {{closeTries}} and {{retryInterval}} property, however this
only retries the rename operation, not the close operation as seen in the documentation:
> {quote}The file may still remain open if the close call fails but the data will be intact
and in this case, the file will be closed only after a Flume restart.
> {quote}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message