spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From viirya <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-3327] Make broadcasted value mutable fo...
Date Wed, 03 Sep 2014 13:20:24 GMT
Github user viirya commented on the pull request:

    https://github.com/apache/spark/pull/2217#issuecomment-54295028
  
    @srowen Thanks for comment. In fact I want some persistent mutable states per data partition.
I just achieve that goal with mutable broadcasted variables. I know that a broadcasted variable
would be shared across many tasks. So I do use a Map variable to separate different variables
for different data partition since I can differentiate them with partition index given by
mapPartitionWithIndex.
    
    It might look at odds with Spark's design then. However, if mutable Broadcast variable
is a bad idea, can we have a mechanism to have persistent mutable states per data partition?
Or it is a bad idea too for Spark's design?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message