spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SLiZn Liu <sliznmail...@gmail.com>
Subject Re: value toDF is not a member of RDD object
Date Wed, 13 May 2015 03:28:54 GMT
Thanks folks, really appreciate all your replies! I tried each of your
suggestions and in particular, *Animesh*‘s second suggestion of *making
case class definition global* helped me getting off the trap.

Plus, I should have paste my entire code with this mail to help the
diagnose.

REGARDS,
Todd Leo
​

On Wed, May 13, 2015 at 12:10 AM Dean Wampler <deanwampler@gmail.com> wrote:

> It's the import statement Olivier showed that makes the method available.
>
> Note that you can also use `sc.createDataFrame(myRDD)`, without the need
> for the import statement. I personally prefer this approach.
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Tue, May 12, 2015 at 9:33 AM, Olivier Girardot <ssaboum@gmail.com>
> wrote:
>
>> you need to instantiate a SQLContext :
>> val sc : SparkContext = ...
>> val sqlContext = new SQLContext(sc)
>> import sqlContext.implicits._
>>
>> Le mar. 12 mai 2015 à 12:29, SLiZn Liu <sliznmailbox@gmail.com> a écrit :
>>
>>> I added `libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %
>>> "1.3.1"` to `build.sbt` but the error remains. Do I need to import modules
>>> other than `import org.apache.spark.sql.{ Row, SQLContext }`?
>>>
>>> On Tue, May 12, 2015 at 5:56 PM Olivier Girardot <ssaboum@gmail.com>
>>> wrote:
>>>
>>>> toDF is part of spark SQL so you need Spark SQL dependency + import
>>>> sqlContext.implicits._ to get the toDF method.
>>>>
>>>> Regards,
>>>>
>>>> Olivier.
>>>>
>>>> Le mar. 12 mai 2015 à 11:36, SLiZn Liu <sliznmailbox@gmail.com> a
>>>> écrit :
>>>>
>>>>> Hi User Group,
>>>>>
>>>>> I’m trying to reproduce the example on Spark SQL Programming Guide
>>>>> <https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection>,
>>>>> and got a compile error when packaging with sbt:
>>>>>
>>>>> [error] myfile.scala:30: value toDF is not a member of org.apache.spark.rdd.RDD[Person]
>>>>> [error] val people = sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p
=> Person(p(0), p(1).trim.toInt)).toDF()
>>>>> [error]                                                             
                                                                ^
>>>>> [error] one error found
>>>>> [error] (compile:compileIncremental) Compilation failed
>>>>> [error] Total time: 3 s, completed May 12, 2015 4:11:53 PM
>>>>>
>>>>> I double checked my code includes import sqlContext.implicits._ after
>>>>> reading this post
>>>>> <https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3C1426522113299-22083.post@n3.nabble.com%3E>
>>>>> on spark mailing list, even tried to use toDF("col1", "col2")
>>>>> suggested by Xiangrui Meng in that post and got the same error.
>>>>>
>>>>> The Spark version is specified in build.sbt file as follows:
>>>>>
>>>>> scalaVersion := "2.11.6"
>>>>> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.3.1"
% "provided"
>>>>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "1.3.1"
>>>>>
>>>>> Anyone have ideas the cause of this error?
>>>>>
>>>>> REGARDS,
>>>>> Todd Leo
>>>>> ​
>>>>>
>>>>
>

Mime
View raw message