flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Baris Akgun (Garanti Teknoloji)" <BarisA...@garanti.com.tr>
Subject Flume kafka channel
Date Fri, 08 Apr 2016 07:44:59 GMT
Hi

I am using kafka as a channel in my flume conf file.

How can I set  serializer.class" with "kafka.serializer.StringEncoder" value in flume confugration
file? Or How can I set flume's kafka consumer encoder?

Thanks.

My Conf File

tier1.sinks = sink1
tier1.channels = channel1
#
tier1.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannel
tier1.channels.channel1.capacity=1000000
tier1.channels.channel1.transactionCapacity=100000
tier1.channels.channel1.brokerList=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
tier1.channels.channel1.topic=topic_baris2
tier1.channels.channel1.zookeeperConnect=yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
tier1.channels.channel1.parseAsFlumeEvent=false
#
tier1.sinks.sink1.type = hdfs
tier1.sinks.sink1.hdfs.useLocalTimeStamp=true
tier1.sinks.sink1.hdfs.path = /data/stage/flume/social_media/tweet_kafka/%Y%m%d
tier1.sinks.sink1.hdfs.filePrefix = events_%Y%m%d
tier1.sinks.sink1.hdfs.fileSuffix = .log
tier1.sinks.sink1.hdfs.inUsePrefix = .
tier1.sinks.sink1.hdfs.inUseSuffix = .incomplete
tier1.sinks.sink1.hdfs.round = true
tier1.sinks.sink1.hdfs.roundValue = 1
tier1.sinks.sink1.hdfs.roundUnit = minute
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.rollSize = 268435456
tier1.sinks.sink1.hdfs.batchSize = 100000
tier1.sinks.sink1.hdfs.fileType=DataStream
tier1.sinks.sink1.hdfs.writeFormat=Text
#
tier1.sinks.sink1.channel = channel1


Barış
Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu
mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi
ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve
eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin
herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige
ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden
ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated
in this message. If you received this message although you are not the addressee, you are
responsible to keep the message confidential. The sender has no responsibility for the accuracy
or correctness of the information in the message and its attachments. Our company shall have
no liability for any changes or late receiving, loss of integrity and confidentiality, viruses
and any damages caused in anyway to your computer system.

Mime
View raw message