beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reinier Kip (JIRA)" <>
Subject [jira] [Created] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover
Date Thu, 31 Aug 2017 11:54:00 GMT
Reinier Kip created BEAM-2831:

             Summary: Possible bug in Beam+Flink memory management, disk spillover
                 Key: BEAM-2831
             Project: Beam
          Issue Type: Bug
          Components: runner-flink
    Affects Versions: 2.1.0, 2.0.0
         Environment: Flink 1.2.1 and 1.3.0, Java 8, macOS 10.12.6 and RedHat 6
            Reporter: Reinier Kip
            Assignee: Aljoscha Krettek

I’ve been running a Beam pipeline on Flink. Depending on the dataset size and the heap memory
configuration of the jobmanager and taskmanager, I may run into an EOFException, which causes
the job to fail.

As [discussed on Flink's mailinglist|]
(stacktrace enclosed), Flink catches these EOFExceptions and activates disk spillover. Because
Beam wraps these exceptions, this mechanism fails, the exception travels up the stack, and
the job aborts.

EOFException occurs with fairly small datasets (tens of megabytes), 768MiB of Flink heap memory,
Beam 2.0.0/2.1.0, Flink 1.2.1/1.3.0, Java 8.

Hopefully this is enough information and this is something that can be adjusted for in Beam.
I'd be glad to provide more information where needed.

This message was sent by Atlassian JIRA

View raw message