flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Betancourt <michelbetanco...@gmail.com>
Subject Fwd: Flink - Exception Handling best practices
Date Fri, 11 Nov 2016 01:33:13 GMT
Hi, new to Apache Flink.  Trying to find some solid input on how best to
handle exceptions in streams -- specifically those that should not
interrupt the stream.

For example, if an error occurs during deserialization from bytes/Strings
to your data-type, in my use-case I would rather queue the data for visual
inspection than discard it and filter it out.

One way of doing this is to diverge the stream so that good items take one
path, while bad items take another.

The closest thing I can find in Flink that can achieve this effect is the
split operator. The caveat is that split does not also allow for inlined
transformations.  In other words, the best use of split appears first
perform your logic that catches the exception.  Then pass the exception
into the next stage which uses split to check for an exception and
providing names to each piece of the decision, for example "OK" vs "error".

Frameworks like RX (Reactive Extensions, eg RxJava) have built in
functionality that allows the user to decide if they want to handle
exceptions globally or specifically and resume if needed.  I was hoping to
find similar operations in Flink but so far no luck.

At any rate, it would be great to get some feedback to see if I am heading
down the good path here, and whether there are any caveats / gotchas to be
aware of?

Thanks!
Mich

Mime
View raw message