hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert B Hamilton <robert.hamil...@gm.com>
Subject Flume rollback during restart possible?
Date Mon, 08 Jun 2015 23:04:27 GMT
Hello all. I have an interesting case where we lose data in the event of a flume crash, which
is easily reproducible when we kill -9  the flume agent.

 I believe that this may be because the Flume Sink is issuing a commit before it actually
completes the fs sync.  If this is the case then the last few commits just before the crash
would have removed events from the queue even though those events will needed to perform a
recovery.  My question is, are those events still possibly in the WAL? If so, is it possible
so somehow roll back the queue to a point in time before the commits were processed, and restart
from that state? How would I accomplish this?

Nothing in this message is intended to constitute an electronic signature unless a specific
statement to the contrary is included in this message.

Confidentiality Note: This message is intended only for the person or entity to which it is
addressed. It may contain confidential and/or privileged material. Any review, transmission,
dissemination or other use, or taking of any action in reliance upon this message by persons
or entities other than the intended recipient is prohibited and may be unlawful. If you received
this message in error, please contact the sender and delete it from your computer.

View raw message