flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Preston Roy <Preston....@telus.com>
Subject Flume use case - send only new file contents
Date Thu, 08 Jun 2017 20:24:03 GMT
Hello all,

I am a very new user to Flume and am unsure if this is an appropriate use of the Flume community.
I am having a minor issue configuring Flume for a particular use case and was hoping you could

I have two Virtual Machines (VMs) connected to each other through an internal network on Oracle
Virtual Box. My goal is to have one VM watch a particular directory that will only ever have
one file in it. When the file is changed, I wish for Flume to only send only the new lines/data.
I want the other VM to receive this data and update/concatenate the data to a single file
in a particular directory on it.

So far, I have this process very close to working. Whenever changes are made in VM1, they
are updated on VM2. However, the entire file on VM1 is sent to VM2 every time, not the new
lines. For example, if I wrote "Test1" and then a while later underneath wrote "Test2" to
the file on VM1, on VM2 the output would be:


What I want to see is:


I am not sure how to implement this, and am sending this email after thoroughly examining
the Flume user guide documentation and most relevant articles on stackoverflow/stackexchange.
For your reference, below are screenshots of the current configuration files (they are working
in the manner I mentioned above).

VM1 configuration:


VM2 configuration:

I realize another solution would be to keep the configuration on VM1 and overwrite the file
on VM2 everytime new contents are detected. However, I am also unsure how to implement this.

Any assistance you could provide is greatly appreciated!

Best regards,

Preston Roy, BSc in Electrical Engineering
Co-op student
Network and Video Services
M +1 (587) 228 8520
Member of the TELUS team
the future is friendly(r)

View raw message