flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Sharma <saurabh.sha...@nviz.com>
Subject filename not appearing in kafka topic event header
Date Mon, 25 Jan 2016 12:10:00 GMT
Hi,

I am trying to configure SpoolingDirectory as a source and Kafka as a sink in apache flume
with following configuration.
Could you please help me to understand how I can send fileheader in an event header to Kafka
topic. Am I missing anything in following configuration.


agent.sources = spoolDir
agent.channels = memoryChannel
agent.sinks = sink
agent.sources.spoolDir.interceptors = i1

#Source configuration
agent.sources.spoolDir.type = spooldir
agent.sources.spoolDir.spoolDir = D:/CommerceSense/spoolDir
agent.sources.spoolDir.fileHeader = true
agent.sources.spoolDir.basenameHeader = true
agent.sources.spoolDir.deserializer = LINE
agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.spoolDir.interceptors.i1.preserveExisting = true
agent.sources.spoolDir.interceptors.i1.prefix = test
agent.sources.spoolDir.channels = memoryChannel


#Channel Configuration
agent.channels.memoryChannel.type = memory


#Sink Configuration
agent.sinks.sink.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.sink.topic = cdnLogsTopic
agent.sinks.sink.brokerList = localhost:9092
agent.sinks.sink.batchSize = 100
agent.sinks.sink.channel = memoryChannel

Mime
View raw message