flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: Unit testing a Kafka stream based application?
Date Thu, 27 Oct 2016 08:07:56 GMT
Thanks.

This is exactly what I needed.

Niels

On Wed, Oct 26, 2016 at 11:03 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> Hi Niels,
>
> Sorry for the late response.
>
> you can launch a Kafka Broker within a JVM and use it for testing purposes.
> Flink's Kafka connector is using that a lot for integration tests. Here is
> the code starting the Kafka server: https://github.com/apache/flink/blob/
> 770f2f83a81b2810aff171b2f56390ef686f725a/flink-streaming-
> connectors/flink-connector-kafka-0.8/src/test/java/org/
> apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java#L358
>
>
> In our tests we also start a Flink mini cluster, to submit jobs to.
> Here is an example: https://github.com/apache/flink/blob/
> f46ca39188dce1764ee6615eb6697588fdc04a2a/flink-streaming-
> connectors/flink-connector-kafka-base/src/test/java/org/
> apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java#L1408
>
> I hope you can find the right code lines to copy for your purposes.
>
> Regards,
> Robert
>
>
> On Fri, Oct 21, 2016 at 4:00 PM, Niels Basjes <Niels@basjes.nl> wrote:
>
>> Hi,
>>
>> In addition to having unit tests for the individual components (map,
>> flatmap, reduce, etc) of my application I would like to write unit tests
>> for the entire flow of my Flink application.
>>
>> My application reads from Kafka, does various processing and writes out
>> put to both kafka and files.
>>
>> This means I need a controlled (mock) Kafka broker where I can insert a
>> specific sequence of messages, my application that reads from that and then
>> writes the output somewhere else.
>>
>> What is the recommended way of doing this?
>> What tools/example projects are available for building such tests?
>>
>> Thanks.
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
View raw message