flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alam, Zeeshan" <Zeeshan.A...@fmr.com>
Subject Flink Batch Processing with Kafka
Date Wed, 03 Aug 2016 12:21:46 GMT
Hi,

Flink works very well with Kafka if you wish to stream data. Following  is how I am streaming
data with Kafka and Flink.

FlinkKafkaConsumer08<Event> kafkaConsumer = new FlinkKafkaConsumer08<>(KAFKA_AVRO_TOPIC,
avroSchema, properties);
DataStream<Event> messageStream = env.addSource(kafkaConsumer);

Is there a way to do a micro batch operation on the data coming from Kafka? What I want to
do is to reduce or aggregate the events coming from Kafka. For instance I am getting 40000
events per second from Kafka and what I want is to group 2000 events into one and send it
to my microservice for further processing. Can I use the Flink DataSet API for this or should
I go with Spark or some other framework?

Thanks & Regards
Zeeshan Alam

Mime
View raw message