flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Re: best way to make all hdfs records in one file under a folder?
Date Mon, 20 Jan 2014 18:57:27 GMT
How is your data partitioned, by date?

On Monday, January 20, 2014, Chen Wang <chen.apache.solr@gmail.com> wrote:

> Guys,
> I have flume setup to flow partitioned data to hdfs, each partition has
> its own file folder. Is there a way to specify all the data under one
> partition to be in one file?
> I am currently using
> MyAgent.sinks.HDFS.hdfs.batchSize = 10000
> MyAgent.sinks.HDFS.hdfs.rollSize = 15000000
> MyAgent.sinks.HDFS.hdfs.rollCount = 10000
> MyAgent.sinks.HDFS.hdfs.rollInterval = 360
>
> to make the file roll on 15m data or after 6 minute.
>
> Is this the best way to achieve my goal?
> Thanks,
> Chen
>
>

Mime
View raw message