spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Updated] (SPARK-5077) Map output statuses can still exceed spark.akka.frameSize
Date Mon, 04 May 2015 11:46:15 GMT


Sean Owen updated SPARK-5077:
    Target Version/s:   (was: 1.2.1, 1.3.0)

> Map output statuses can still exceed spark.akka.frameSize
> ---------------------------------------------------------
>                 Key: SPARK-5077
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 1.2.0, 1.3.0
>            Reporter: Josh Rosen
> Since HighlyCompressedMapOutputStatuses uses a bitmap for tracking empty blocks, its
size is not bounded and thus Spark is still susceptible to "MapOutputTrackerMasterActor: Map
output statuses
> were 11141547 bytes which exceeds spark.akka.frameSize"-type errors, even in 1.2.0.
> We needed to use a bitmap for tracking zero-sized blocks (see SPARK-3740; this isn't
just a performance issue; it's necessary for correctness).  This will require a bit more effort
to fix, since we'll either have to find a way to use a fixed size / capped size encoding for
MapOutputStatuses (which might require changes to let us fetch empty blocks safely) or figure
out some other strategy for shipping these statues.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message