flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nali, Ramgopal (Nokia - IN/Bangalore)" <ramgopal.n...@nokia.com>
Subject Schema registry support for flume
Date Mon, 07 Aug 2017 04:43:36 GMT
Will Flume support Avro Schema Registry?
I have table data coming to kafka topic as Avro. Flume reads from kafka topic and writes to
HDFS as Parquet, later on the hive external tables are created.
I could see that the data in hdfs is created with .tmp extension and not creating a proper
file. For some reason the flume sink is not calling the close on the file, I feel. Following
are the configurations that I am using.

agent.channels = c1
agent.sources = s1
agent.sinks = k1

agent.channels.c1.type = memory
agent.channels.c1.capacity = 10000000
agent.channels.c1.transactionCapacity = 1000

agent.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.s1.channels = c1
agent.sources.s1.kafka.bootstrap.servers=XXXX:1025 XXXX:1025 XXXX:1025
agent.sources.s1.kafka.topics=t10

agent.sinks.k1.channel = c1
agent.sinks.k1.type = org.apache.flume.sink.kite.DatasetSink
agent.sinks.k1.kite.dataset.uri = dataset:hdfs://namenodeHA/kite/avro_income_band_1
#agent.sinks.k1.kite.dataset.uri = dataset:hive:hdfs://namenodeHA/kite/avro_income_bandt1/schema_parquet/income_band
agent.sinks.k1.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder
agent.sinks.k1.hdfs.filePrefix = parquetdata
agent.sinks.k1.hdfs.fileSuffix = .parquet
agent.sinks.k1.hdfs.fileType=DataStream
agent.sinks.k1.serializer.class = kafka.serializer.DefaultEncoder
agent.sinks.k1.kafka.message.coder.schema.registry.class = com.linkedin.camus.schemaregistry.AvroRestSchemaRegistry
agent.sinks.k1.schema.registry.url = http://schema_registry:8081/subjects/income_band/versions/latest
agent.sinks.k1.kite.batchSize = 2
agent.sinks.k1.kite.rollInterval = 30
agent.sinks.k1.kite.flushable.commitOnBatch = true

agent.sources.s1.channels = c1
agent.sinks.k1.channel= c1




Mime
View raw message