flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Halfon, Roey" <roey.hal...@sap.com>
Subject Tracking message processing in my application
Date Thu, 04 Jul 2019 12:29:02 GMT
We are looking for a monitoring solution for our dataflow - Track the progress of incoming
messages while they are processed.
I'll clarify - we want to build some service which will show status for each incoming message.
And in case of failures to give some detailed information.

I thought about the following:
First, every incoming message will be assigned with some id.
We can create a "Reporter" (A logger with some additional capabilities)  which each operator
can communicate with, and update a status and more relevant information. These details can
be stroed in kibana (ES) for example.
Then, we need to create another service which will query kibana and shows the results.

What do you think about that? Is there any built-in solution for that? (flink built in metrics
are not relevant here because they don't help to track a single message)

How are you logging and tracking your processed messages?
Is there any documentation or some use cases that I can learn from?


View raw message