beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (Jira)" <j...@apache.org>
Subject [jira] [Updated] (BEAM-5775) Make the spark runner not serialize data unless spark is spilling to disk
Date Tue, 02 Jun 2020 04:01:07 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kenneth Knowles updated BEAM-5775:
----------------------------------
    Labels: stale-assigned  (was: )

> Make the spark runner not serialize data unless spark is spilling to disk
> -------------------------------------------------------------------------
>
>                 Key: BEAM-5775
>                 URL: https://issues.apache.org/jira/browse/BEAM-5775
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Mike Kaplinskiy
>            Assignee: Mike Kaplinskiy
>            Priority: P3
>              Labels: stale-assigned
>          Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> Currently for storage level MEMORY_ONLY, Beam does not coder-ify the data. This lets
Spark keep the data in memory avoiding the serialization round trip. Unfortunately the logic
is fairly coarse - as soon as you switch to MEMORY_AND_DISK, Beam coder-ifys the data even
though Spark might have chosen to keep the data in memory, incurring the serialization overhead.
>  
> Ideally Beam would serialize the data lazily - as Spark chooses to spill to disk. This
would be a change in behavior when using beam, but luckily Spark has a solution for folks
that want data serialized in memory - MEMORY_AND_DISK_SER will keep the data serialized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message