Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Subject: Re: Error handling
From: Aljoscha Krettek <aljoscha@apache.org>
In-Reply-To: 
 <CANC1h_sNhWF_iapXqf-W3aKWav1==_4WiS4iH6teeFv-hT7njQ@mail.gmail.com>
Date: Mon, 16 Nov 2015 19:27:08 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <93EAA3A7-B0CF-4137-86D6-E95C9B5CDBF3@apache.org>
References: 
 <CANZa=GvfFxayAP1TG8U85TAY+4=6oj8he0SyGXg1Ys5cELY2rw@mail.gmail.com>
 <A6A319FB-0EF2-4FF3-A9C3-9F4D90967CCD@apache.org>
 <CANZa=Gsa2TW1YhkP3-n_YYq0VkB2yZCvLYfiAQ=XzpQT03h_aQ@mail.gmail.com>
 <CANC1h_sNhWF_iapXqf-W3aKWav1==_4WiS4iH6teeFv-hT7njQ@mail.gmail.com>
To: user@flink.apache.org

Hi,
I don=E2=80=99t think that alleviates the problem. Sometimes you might =
want the system to continue even if stuff outside the UDF fails. For =
example, if a serializer does not work because of a null value =
somewhere. You would, however, like to get a message about this =
somewhere, I assume.

Cheers,
Aljoscha
> On 16 Nov 2015, at 19:22, Stephan Ewen <sewen@apache.org> wrote:
>=20
> Hi Nick!
>=20
> The errors outside your UDF (such as network problems) will be handled =
by Flink and cause the job to go into recovery. They should be =
transparently handled.
>=20
> Just make sure you activate checkpointing for your job!
>=20
> Stephan
>=20
>=20
> On Mon, Nov 16, 2015 at 6:18 PM, Nick Dimiduk <ndimiduk@gmail.com> =
wrote:
> I have been thinking about this, maybe we can add a special output =
stream (for example Kafka, but can be generic) that would get =
errors/exceptions that where throws during processing. The actual =
processing would not stop and the messages in this special stream would =
contain some information about the current state of processing, the =
input element(s) and the machine/VM where computation is happening.
>=20
> Yes, this is precisely what I have in mind. The goal is (1) to not =
lose input data, and (2) to make errors available for operator =
visibility.
>=20
> It's not very portable, but I was able to implement my Maybe<IN, OUT, =
Throwable> type. I can now use it as the output of all my source =
streams, and split those streams on the presence of the Throwable. With =
this, I'm able to trap certain forms of invalid input and send it to an =
errors sink. However, there are still some error cases that cause =
exceptions, apparently, outside of my UDF try block that cause the whole =
streaming job to terminate.
> =20
> > On 11 Nov 2015, at 21:49, Nick Dimiduk <ndimiduk@gmail.com> wrote:
> >
> > Heya,
> >
> > I don't see a section in the online manual dedicated to this topic, =
so I want to raise the question here: How should errors be handled? =
Specifically I'm thinking about streaming jobs, which are expected to =
"never go down". For example, errors can be raised at the point where =
objects are serialized to/from sources/sinks, and UDFs. Cascading =
provides failure traps [0] where erroneous tuples are saved off for =
post-processing. Is there any such functionality in Flink?
> >
> > I started down the road of implementing a Maybe/Optional type, a =
POJO Generic triple of <IN, OUT, Throwable> for capturing errors at each =
stage of a pipeline. However, Java type erasure means even though it =
compiles, the job is rejected at submission time.
> >
> > How are other people handling errors in their stream processing?
> >
> > Thanks,
> > Nick
> >
> > [0]: =
http://docs.cascading.org/cascading/1.2/userguide/html/ch06s03.html
>=20
>=20
>=20