flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Ma <henry.ma.1...@gmail.com>
Subject Reliability in Flume
Date Thu, 24 Jan 2013 03:26:13 GMT
Dear Flume developers and users,

I understand that Flume NG uses channel-based transactions to guarantee
reliable message delivery between agents. But in some extreme failure
scenes, will Flume keep total Reliability? I have thought of these scenes
below.

1. In transactions between agent, what will happen if the receiving agent
process down just after it commits its put transaction and before sends the
success indication to the sending agent? Will the sending agent send the
same event again when the receiving agent recovers, and cause data
duplication?

2. In the communication between the client (data source, sending data to
the first-hop agent) and the  first-hop agent, what will happen if the
agent process down just after it receives the event and before saves to its
channel? Will it cause data loss?

3. In the communication between the final-hup agent and the storage system
(such as MySQL, HDFS, file system, etc.), what happened if the agent down
before it commits the saving transaction but has saved some data in the
storage? Will this cause data duplication after the recover of the agent?

Thank you very much!
-- 
Best Regards,
Henry Ma

Mime
View raw message