flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Percy <mpe...@apache.org>
Subject Re: .tmp in hdfs sink
Date Thu, 15 Nov 2012 19:54:22 GMT
Hi Mohit, this is a complicated issue. I've filed
https://issues.apache.org/jira/browse/FLUME-1714 to track it.

In short, it would require a non-trivial amount of work to implement this,
and it would need to be done carefully. I agree that it would be better if
Flume handled this case more gracefully than it does today. Today, Flume
assumes that you have some job that would go and clean up the .tmp files as
needed, and that you understand that they could be partially written if a
crash occurred.

Regards,
Mike

On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:

> What we are seeing is that if flume gets killed either because of server
> failure or other reasons, it keeps around the .tmp file. Sometimes for
> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
> file more gracefully?
>

Mime
View raw message