flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Smirnov <alexander.smirn...@gmail.com>
Subject Re: Tracking deserialization errors
Date Mon, 23 Apr 2018 09:59:43 GMT
That's absolutely no problem Tzu-Li. Either of them would work. Thank you!

On Thu, Apr 19, 2018 at 4:56 PM Tzu-Li (Gordon) Tai <tzulitai@apache.org>

> @Alexander
> Sorry about that, that would be my mistake. I’ll close FLINK-9204 as a
> duplicate and leave my thoughts on FLINK-9155. Thanks for pointing out!
> On 19 April 2018 at 2:00:51 AM, Elias Levy (fearsome.lucidity@gmail.com)
> wrote:
> Either proposal would work.  In the later case, at a minimum we'd need a
> way to identify the source within the metric.  The basic error metric would
> then allow us to go into the logs to determine the cause of the error, as
> we already record the message causing trouble in the log.
> On Mon, Apr 16, 2018 at 4:42 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>> Thanks for starting the discussion Elias.
>> I see two ways to address this issue.
>> 1) Add an interface that a deserialization schema can implement to
>> register metrics. Each source would need to check for the interface and
>> call it to setup metrics.
>> 2) Check for null returns in the source functions and increment a
>> respective counter.
>> In both cases, we need to touch the source connectors.
>> I see that passing information such as topic name, partition, and offset
>> are important debugging information. However, I don't think that metrics
>> would be good to capture them.
>> In that case, log files might be a better approach.
>> I'm not sure to what extend the source functions (Kafka, Kinesis) support
>> such error tracking.
>> Adding Gordon to the thread who knows the internals of the connectors.

View raw message