flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vignesh <vignesh...@gmail.com>
Subject Flume FTP Source - Configure Dynamic directory name
Date Wed, 08 Nov 2017 15:48:43 GMT
Hi,

I would get a requestID from HTTP source and based on that requestID i
will have to form the directory structure/name. Under that directory
there would be many files available in a remote server and need to
fetch all the files from that directory and has to be put in HDFS. We
would not be able to run flume agent in that remote server. So i took
help of flume-ftp-source.

The problem now is that how do i pass a requestID which i get from
HTTP source to custom flume-ftp-source and form the directory
name(parameter:). This seems to be a

HTTP source -> Custom FTP Source -> HDFS Sink

problem. But i understand that two sources could not be connected. Is
there any other way i could accomplish this.

Thanks and Regards,
Vignesh

Mime
View raw message