spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-5077) Map output statuses can still exceed spark.akka.frameSize
Date Mon, 04 May 2015 11:46:15 GMT

     [ https://issues.apache.org/jira/browse/SPARK-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-5077:
-----------------------------
    Target Version/s:   (was: 1.2.1, 1.3.0)

> Map output statuses can still exceed spark.akka.frameSize
> ---------------------------------------------------------
>
>                 Key: SPARK-5077
>                 URL: https://issues.apache.org/jira/browse/SPARK-5077
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 1.2.0, 1.3.0
>            Reporter: Josh Rosen
>
> Since HighlyCompressedMapOutputStatuses uses a bitmap for tracking empty blocks, its
size is not bounded and thus Spark is still susceptible to "MapOutputTrackerMasterActor: Map
output statuses
> were 11141547 bytes which exceeds spark.akka.frameSize"-type errors, even in 1.2.0.
> We needed to use a bitmap for tracking zero-sized blocks (see SPARK-3740; this isn't
just a performance issue; it's necessary for correctness).  This will require a bit more effort
to fix, since we'll either have to find a way to use a fixed size / capped size encoding for
MapOutputStatuses (which might require changes to let us fetch empty blocks safely) or figure
out some other strategy for shipping these statues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message