beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reinier Kip (JIRA)" <>
Subject [jira] [Commented] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover
Date Thu, 31 Aug 2017 19:49:00 GMT


Reinier Kip commented on BEAM-2831:

[~aljoscha] Yes! The pipeline consistently succeeds with this change. Thanks for your effort.
I suspect we will intregrate this replacement coder while we wait for the fix to come out.
Do you foresee any consequences due to the considerations you wrote near the bottom?

> Possible bug in Beam+Flink memory management, disk spillover
> ------------------------------------------------------------
>                 Key: BEAM-2831
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.0.0, 2.1.0
>         Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 10.12.6
and unknown Linux
>            Reporter: Reinier Kip
>            Assignee: Aljoscha Krettek
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and the heap
memory configuration of the jobmanager and taskmanager, I may run into an EOFException, which
causes the job to fail.
> As [discussed on Flink's mailinglist|]
(stacktrace enclosed), Flink catches these EOFExceptions and activates disk spillover. Because
Beam wraps these exceptions, this mechanism fails, the exception travels up the stack, and
the job aborts.
> Hopefully this is enough information and this is something that can be adjusted for in
Beam. I'd be glad to provide more information where needed.

This message was sent by Atlassian JIRA

View raw message