spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-2585) Remove special handling of Hadoop JobConf
Date Thu, 20 Nov 2014 07:18:33 GMT


Josh Rosen resolved SPARK-2585.
    Resolution: Won't Fix

I'm going to resolve this as "Won't Fix" since every attempt at removing this special handling
has caused serious performance regressions.  Feel free to re-open this if we discover a workaround
that doesn't cause a huge slowdown while deserializing configurations.

> Remove special handling of Hadoop JobConf
> -----------------------------------------
>                 Key: SPARK-2585
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Patrick Wendell
>            Assignee: Josh Rosen
>            Priority: Critical
> This is a follow up to SPARK-2521 and should close SPARK-2546 (provided the implementation
does not use shared conf objects). We no longer need to specially broadcast the Hadoop configuration
since we are broadcasting RDD data anyways.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message