spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh <ashutosh.triv...@iiitb.org>
Subject Re: [MLlib] Contributing Algorithm for Outlier Detection
Date Tue, 18 Nov 2014 10:40:14 GMT
Hi Anant,


I have removed the counter and all possible side effects. Now I think we can go ahead with
the testing. I have created another folder for testing. I will add you as a collaborator in
github .


_Ashutosh

________________________________
From: slcclimber [via Apache Spark Developers List] <ml-node+s1001551n9399h45@n3.nabble.com>
Sent: Monday, November 17, 2014 10:45 AM
To: Ashutosh Trivedi (MT2013030)
Subject: Re: [MLlib] Contributing Algorithm for Outlier Detection

Ashutosh,
The counter will certainly be an parellization issue when multiple nodes are used specially
over massive datasets.
A better approach would be to use some thing along these lines:

    val index = sc.parallelize(Range.Long(0, rdd.count, 1), rdd.partitions.size)
    val rddWithIndex = rdd.zip(index)
Which zips the two RDD's in a parallelizable fashion.


________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-developers-list.1001551.n3.nabble.com/MLlib-Contributing-Algorithm-for-Outlier-Detection-tp8880p9399.html
To unsubscribe from [MLlib] Contributing Algorithm for Outlier Detection, click here<http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=8880&code=YXNodXRvc2gudHJpdmVkaUBpaWl0Yi5vcmd8ODg4MHwtMzkzMzE5NzYx>.
NAML<http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/MLlib-Contributing-Algorithm-for-Outlier-Detection-tp8880p9420.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message