beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Commented] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover
Date Fri, 01 Sep 2017 21:21:01 GMT


Kenneth Knowles commented on BEAM-2831:

I think for the most part there's no implications. I have no immediate objection based on
any principles. The choice between wrapping an IOException into a CoderException versus not
doing so is not clear in many cases. In particular, EOF and parse error are not really distinct
errors, since malformed data causes the EOF.

This may violate our rigid backwards compatibility requirements, though it probably affects
zero actual users. Especially since they have to handle all IOExceptions anyhow, and because
of the semantic ambiguity above, basically have to take the same action too.

We have 100% rigidity on type errors and leaving working code working, but for bugs and places
where the behavior is not well-defined (like what kind of exception a particular coder throws
in particular situations) maybe there's flexibility...

> Possible bug in Beam+Flink memory management, disk spillover
> ------------------------------------------------------------
>                 Key: BEAM-2831
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.0.0, 2.1.0
>         Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 10.12.6
and unknown Linux
>            Reporter: Reinier Kip
>            Assignee: Aljoscha Krettek
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and the heap
memory configuration of the jobmanager and taskmanager, I may run into an EOFException, which
causes the job to fail.
> As [discussed on Flink's mailinglist|]
(stacktrace enclosed), Flink catches these EOFExceptions and activates disk spillover. Because
Beam wraps these exceptions, this mechanism fails, the exception travels up the stack, and
the job aborts.
> Hopefully this is enough information and this is something that can be adjusted for in
Beam. I'd be glad to provide more information where needed.

This message was sent by Atlassian JIRA

View raw message