spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: Text
Date Fri, 27 Jan 2017 13:15:17 GMT
I would not count on order preserving nature of the operations, because it
is not guranteed. I would assign some order to the sentences and sort at
the end before write back

On Fri, 27 Jan 2017 at 10:59 pm, Md. Rezaul Karim <
rezaul.karim@insight-centre.org> wrote:

> Some operations like map, filter, flatMap and coalesce (with
> shuffle=false) usually preserve the order. However, sortBy, reduceBy,
> partitionBy, join etc. do not.
>
> Regards,
> _________________________________
> *Md. Rezaul Karim*, BSc, MSc
> PhD Researcher, INSIGHT Centre for Data Analytics
> National University of Ireland, Galway
> IDA Business Park, Dangan, Galway, Ireland
> Web: http://www.reza-analytics.eu/index.html
> <http://139.59.184.114/index.html>
>
> On 27 January 2017 at 09:44, Soheila S. <soheila518@gmail.com> wrote:
>
> Hi All,
> I read a test file using sparkContext.textfile(filename) and assign it to
> an RDD and process the RDD (replace some words) and finally write it to
> a text file using rdd.saveAsTextFile(output).
> Is there any way to be sure the order of the sentences will not be
> changed? I need to have the same text with some corrected words.
>
> thanks!
>
> Soheila
>
>
> --
Best Regards,
Ayan Guha

Mime
View raw message