spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: update of RDDs
Date Thu, 01 May 2014 14:10:33 GMT
if you are doing a lot of small updates on a huge amount of data & need to
get real time response on the output spark is probably not a good fit.
If you are dong small updates on your rdd but need to materialize the final
rdd with all the changes every 1 day or so then probably spark can fit with
some data modelling.


Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Thu, May 1, 2014 at 6:31 PM, NN Murthy <NN.Murthy@cmcltd.com> wrote:

> Thanks a lot for very prompt response. Then next questions are the
> following.
>
> 1.       Can we conclude that Spark is NOT the solution for our
> requirement? Or
>
> 2.       Is there a design approach to meet such requirements using
> Spark?
>
>
>
> *From:* Mayur Rustagi [mailto:mayur.rustagi@gmail.com]
> *Sent:* 01 May 2014 18:22
> *To:* user@spark.apache.org
> *Cc:* user@spark.incubator.apache.org
> *Subject:* Re: update of RDDs
>
>
>
> RDD are immutable so cannot be updated. You can create new RDD containing
> updated entries(often not what you want to do).
>
>
>
>
>
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
>
> http://www.sigmoidanalytics.com
>
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
>
>
> On Thu, May 1, 2014 at 4:42 AM, narayanabhatla NarasimhaMurthy <
> NN.Murthy@cmcltd.com> wrote:
>
> In our application, we need distributed RDDs containing key-value maps. We
> have operations that update RDDs by way of adding entries to the map,
> delete
> entries from the map as well as update value part of maps.
> We also have map reduce functions that operate on the RDDs.The questions
> are
> the following.
> 1. Can RDDs be updated? if Yes, what rae the methods?
> 2. If we update RDDs, will it happen in place or does it create new RDDs
> with almost double the original RDD size (original+newly created RDD)?
> Thank you very much.
> N.N.Murthy
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/update-of-RDDs-tp5132.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>
> DISCLAIMER
> The information contained in this e-mail message and/or attachments to it
> may contain confidential or privileged information. If you are not the
> intended recipient, any dissemination, use, review, distribution, printing
> or copying of the information contained in this e-mail message and/or
> attachments to it are strictly prohibited. If you have received this
> communication in error, please notify us by reply e-mail or directly to
> netsupport@cmcltd.com or telephone and immediately and permanently delete
> the message and any attachments. Thank you.
> This email has been scrubbed for your protection by SecureMX. For more
> information visit securemx.in
>

Mime
View raw message