spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiangrui Meng (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1112) When spark.akka.frameSize > 10, task results bigger than 10MiB block execution
Date Thu, 17 Jul 2014 04:02:07 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiangrui Meng updated SPARK-1112:
---------------------------------

    Target Version/s: 0.9.2, 1.0.1, 1.1.0  (was: 1.0.1, 1.1.0, 0.9.3)

> When spark.akka.frameSize > 10, task results bigger than 10MiB block execution
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-1112
>                 URL: https://issues.apache.org/jira/browse/SPARK-1112
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.9.0, 1.0.0
>            Reporter: Guillaume Pitel
>            Assignee: Xiangrui Meng
>            Priority: Blocker
>             Fix For: 1.0.1, 1.1.0
>
>
> When I set the spark.akka.frameSize to something over 10, the messages sent from the
executors to the driver completely block the execution if the message is bigger than 10MiB
and smaller than the frameSize (if it's above the frameSize, it's ok)
> Workaround is to set the spark.akka.frameSize to 10. In this case, since 0.8.1, the blockManager
deal with  the data to be sent. It seems slower than akka direct message though.
> The configuration seems to be correctly read (see actorSystemConfig.txt), so I don't
see where the 10MiB could come from 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message