flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Sinclair <dsincl...@chariotsolutions.com>
Subject HDFS Sink Question
Date Tue, 01 Oct 2013 15:04:03 GMT
Hi all,

I have created an AMQP Source that is being used to feed an HDFS Sink.
Everything is working as expected, but I wanted to try out some error
scenarios.

After creating a file in HDFS and starting to write to it I shutdown HDFS.
I saw the errors in the log as I would expect, and after the configured
roll time tried to close the file. Since HDFS wasn't running it wasn't able
to do so. I restarted HDFS in hope that it would try the close again but it
did not.

Can someone tell me expected behavior under the following scenarios?


   - HDFS isn't available before ever trying to create/write to a file
   - HDFS becomes unavailable after already creating a file and starting to
   write to it
   - HDFS is unavailable when trying to close a file

I'd also be happy to contribute the AMQP source. I wrote the old version
for the original flume

https://github.com/stampy88/flume-amqp-plugin/

Let me know if you'd be interested and thanks for the answers.

dave

Mime
View raw message