flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Krawczyk <ste...@nextdoor.com>
Subject Re: HDFS Sink fails to rename file in Flume 1.4
Date Tue, 03 Feb 2015 23:09:31 GMT
Yes I have some logs, but they don't show any errors as far as I can tell;
in comparison to other files, the log statement that the file is being
renamed is missing.
Do you have something in particular I should look for?

What's the timeline for 1.6 at this stage?



On Tue, Feb 3, 2015 at 2:49 PM, Hari Shreedharan <hshreedharan@cloudera.com>

> Do you have some logs around the time the rename does not happen? We
> recently pushed a workaround for this - when HDFS is unable to close the
> file, we were not renaming it. But since the data is safe (it is hflushed
> anyway), we decided to change that and rename the file anyway:
> https://issues.apache.org/jira/browse/FLUME-2586
> Thanks,
> Hari
> On Tue, Feb 3, 2015 at 2:48 PM, Stefan Krawczyk <stefan@nextdoor.com>
> wrote:
>> Hi Fellow Flume Users,
>> I'm noticing that we get some "stranded" files in S3 that never get
>> renamed from .tmp to .txt.
>> Questions:
>> 1) has this problem been fixed in 1.5, I see some jiras that might
>> address the problem, but none are quite that descriptive for me to figure
>> that out?
>> 2) if this hasn't been addressed, is there a bug for it that I can track?
>> Otherwise any tips or advice appreciated.
>> Cheers,
>> Stefan

View raw message