flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Is Memory Channel data lost on process stop?
Date Wed, 04 Jun 2014 15:39:57 GMT
Thanks for the info, Mike.  Added
https://issues.apache.org/jira/browse/FLUME-2396 to address persisting
in-memory data to disk on (clean) shutdown.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Wed, May 28, 2014 at 5:54 PM, Mike Percy <mpercy@apache.org> wrote:

> The traditional memory channel does not store anything to disk on process
> shutdown, so you lose the data if you kill the process. However, a plain
> reconfiguration will not reinitialize or clear the channel as long as the
> channel name remains the same.
>
> The recoverable memory channel thing (FLUME-896) was never stable and was
> removed some time back.
>
> As far as I know, the spillable memory channel does not today spill to
> disk on shutdown, but one could conceive of adding this feature, since it
> uses its own internal file channel instance to "spill" when its memory
> capacity has been exceeded.
>
> Mike
>
>
> On Wed, May 28, 2014 at 1:44 PM, Christopher Shannon <
> cshannon108@gmail.com> wrote:
>
>> Good question; I look forward to someone cogently answering it.
>>
>> Related, and you probably know this already, the only signals that invoke
>> the JVM shutdown hooks (clean shutdown) are HUP and TERM. (kill -HUP <PID>;
>> kill -TERM <PID>).
>> On May 28, 2014 3:37 PM, "Otis Gospodnetic" <otis.gospodnetic@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> What happens with events buffered in the Memory channel (say via the
>>> Spillable version - https://issues.apache.org/jira/browse/FLUME-1227 or
>>> via its Durable cousin - https://issues.apache.org/jira/browse/FLUME-896
>>> ) when the process is stopped?
>>>
>>> Say the process is stopped gently and appropriately, via an init.d
>>> script or a gentle kill (not kill -9).
>>>
>>> Is the data that was in memory still lost?
>>> Or is there a Shutdown hook that gets run to preserve data on disk?
>>>
>>> Thanks,
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>
>

Mime
View raw message