flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Streaming job monitoring
Date Thu, 08 Jun 2017 09:43:44 GMT
Hi to all,
we've successfully ran our first straming job on a Flink cluster (with some
problems with the shading of guava..) and it really outperforms Logstash,
from the point of view of indexing speed and easiness of use.

However there's only one problem: when the job is running, in the Job
Monitoring UI, I see 2 blocks within the plan visualizer:

   1. Source: Custom File Source (without any info about the file I'm
   reading)
   2. Split Reader: Custom File source -> Sink: unnamed

None of them helps me to understand which data I'm reading or writing
(while within the batch jobs this is usually displayed). Moreover, in the
task details the "Byte sent/Records sent" are totally senseless, I don't
know what is counted (see the attached image if available)...I see
documents indexed on ES but in the Flink Job UI I don't see anything that
could help to understand how many documents are sent to ES or from one
function (Source) to the other (Sink).
I tried to display some metrics and there I found something but I hope this
is not the usual way of monitoring streaming jobs...am I doing something
wrong? Or the streaming jobs should be monitored with something else?

[image: Inline image 1]
Best,
Flavio

Mime
View raw message