flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Restarts without data loss
Date Mon, 09 Jul 2012 08:22:40 GMT
On Mon, Jul 9, 2012 at 8:51 AM, Juhani Connolly <
juhani_connolly@cyberagent.co.jp> wrote:

>   - Intended setup with flume was a file channel connected to an avro
> sink. With only a single disk available, it is extremely slow. JDBC channel
> is also extremely slow, and MemoryChannel will fill up and start refusing
> puts as soon as a network issue comes up.
>

Have you taken a few thread dumps or done other analysis? When you say
"extremely slow" what do you mean? Configured for no dataloss FileChannel
is going to be doing a lot of fsync'ing so I am not surprised it's slow.
That is a property of disks not FileChannel. I think we should use group
commit but that shouldn't make it 10x faster.

Brock

Mime
View raw message