spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From long <longteng...@gmail.com>
Subject Re: Can I add a new method to RDD class?
Date Mon, 05 Dec 2016 10:06:04 GMT
Thank you very much! But why can’t I just add new methods in to the source code of RDD?

> On Dec 5, 2016, at 3:15 AM, Michal Šenkýř [via Apache Spark Developers List] <ml-node+s1001551n20106h87@n3.nabble.com>
wrote:
> 
> A simple Scala example of implicit classes:
> 
> implicit class EnhancedString(str: String) {
>   def prefix(prefix: String) = prefix + str
> }
> 
> println("World".prefix("Hello "))
> As Tarun said, you have to import it if it's not in the same class where you use it.
> 
> Hope this makes it clearer,
> 
> Michal Senkyr
> 
> 
> On 5.12.2016 07:43, Tarun Kumar wrote:
>> Not sure if that's documented in terms of Spark but this is a fairly common pattern
in scala known as "pimp my library" pattern, you can easily find many generic example of using
this pattern.
>> 
>> If you want I can quickly cook up a short conplete example with rdd(although there
is nothing really more to my example in earlier mail) ?
>> 
>> Thanks 
>> Tarun Kumar
>> 
>> On Mon, 5 Dec 2016 at 7:15 AM, long <[hidden email] <x-msg://22/user/SendEmail.jtp?type=node&node=20106&i=0>>
wrote:
>> So is there documentation of this I can refer to? 
>> 
>>> On Dec 5, 2016, at 1:07 AM, Tarun Kumar [via Apache Spark Developers List] <[hidden
email] <http://user/SendEmail.jtp?type=node&node=20104&i=0>> wrote:
>>> 
>> 
>>> Hi Tenglong,
>>> 
>>> In addition to trsell's reply, you can add any method to an rdd without making
changes to spark code.
>>> 
>>> This can be achieved by using implicit class in your own client code:
>>> 
>>> implicit class extendRDD[T](rdd: RDD[T]){
>>> 
>>>      def foo()
>>> 
>>> }
>>> 
>>> Then you basically nees to import this implicit class in scope where you want
to use the new foo method.
>>> 
>>> Thanks
>>> Tarun Kumar 
>>> 
>> 
>>> On Mon, 5 Dec 2016 at 6:59 AM, <<a href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=0"
target="_top" rel="nofollow" link="external" class="">[hidden email]> wrote:
>> 
>>> How does your application fetch the spark dependency? Perhaps list your project
dependencies and check it's using your dev build.
>>> 
>>> 
>> 
>>> On Mon, 5 Dec 2016, 08:47 tenglong, <<a href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=1"
target="_top" rel="nofollow" link="external" class="">[hidden email]> wrote:
>> 
>>> Hi,
>>> 
>>> Apparently, I've already tried adding a new method to RDD,
>>> 
>>> for example,
>>> 
>>> class RDD {
>>>   def foo() // this is the one I added
>>> 
>>>   def map()
>>> 
>>>   def collect()
>>> }
>>> 
>>> I can build Spark successfully, but I can't compile my application code
>>> which calls rdd.foo(), and the error message says
>>> 
>>> value foo is not a member of org.apache.spark.rdd.RDD[String]
>>> 
>>> So I am wondering if there is any mechanism prevents me from doing this or
>>> something I'm doing wrong?
>>> 
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html
<http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100.html>
>>> Sent from the Apache Spark Developers List mailing list archive at Nabble.com
<http://nabble.com/>.
>>> 
>>> ---------------------------------------------------------------------
>> 
>>> To unsubscribe e-mail: <a href="x-msg://19/user/SendEmail.jtp?type=node&amp;node=20102&amp;i=2"
target="_top" rel="nofollow" link="external" class="">[hidden email]
>>> 
>>> 
>>> 
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html
<http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20102.html>
>>> To unsubscribe from Can I add a new method to RDD class?, click here <>.
>>> NAML <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> 
>> View this message in context: Re: Can I add a new method to RDD class? <http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20104.html>
>> Sent from the Apache Spark Developers List mailing list archive <http://apache-spark-developers-list.1001551.n3.nabble.com/>
at Nabble.com.
> 
> 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html
<http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20106.html>
> To unsubscribe from Can I add a new method to RDD class?, click here <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=20100&code=bG9uZ3RlbmcuY3FAZ21haWwuY29tfDIwMTAwfC0xNzQ1MzUzNzE=>.
> NAML <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Can-I-add-a-new-method-to-RDD-class-tp20100p20107.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Mime
View raw message