flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jimmyjack@gmail.com" <jimmyj...@gmail.com>
Subject file sizes in HDFS
Date Thu, 17 Oct 2013 20:41:38 GMT

What are you experiences with file sizes in HDFS when using HDFS sink? The default settings
seem very small. Do you keep the files small and combine them later or do you keep them large
e.g. block size?

I am a bit unsure about keeping large files open for long time (dealing with expiration lease)
and whether flume can correctly close them (and deal with tmp files) in case of a crash.

thank you

View raw message