flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: File Channel Capacity issue
Date Fri, 23 Nov 2012 21:14:25 GMT

The FileChannel actually uses a fixed size checkpoint file -- so it is not possible to set
it to unlimited size (the checkpoint file is mmap-ed to a fixed size buffer). To change the
capacity of the channel, the easiest way off the top of my head is:

* Shutdown the agent.
* Delete all files in the file channel's checkpoint directory. (not the data directories.
Also you might want to move them out, rather than delete to be safe)
* Change your configuration to increase the capacity of the channel.
* Restart the agent - this will cause full replay, so the agent might take sometime to start
up if there are a lot of events in the channel (to avoid this - shutdown the source before
shutting the agent down - so the sink can drain out the channel completely, wait for about
1-2 mins after the channel is empty so that the data files get deleted (this happens only
immediately after a checkpoint - you can verify this by making sure each data dir has only
2 files each), since all events have been sent out - so during restart the channel will be
quite empty, with very little to replay).

Hope this helps.


Hari Shreedharan

On Friday, November 23, 2012 at 11:58 AM, Camp, Roy wrote:

> I am having issue with a slow sink (not flume related) but it is causing my file channel
to overflow.  When trying to increase it I get the following error but can’t seem to find
the setting it refers to in any of the documentation.  Additionally, is there a way to set
the file channel to unlimited?  It doesn’t seem to like 0.  
> java.lang.IllegalStateException: Channel closed [channel=collectorfile]. Due to java.lang.IllegalStateException:
Configured capacity is 100000000 but the  checkpoint file capacity is 1000000. See FileChannel
documentation on how to change a channels capacity.
> Thanks,
> Roy

View raw message